Multiple user interfaces - Cross - platform applications and context - aware interfaces

Multiple User Interfaces Cross-Platform Applications and Context-Aware Interfaces Edited by Ahmed Seffah and Homa Javahery Concordia University, Department of Computer Science, Canada Multiple User Interfaces Multiple User Interfaces Cross-Platform Applications and Context-Aware Interfaces Edited by Ahmed Seffah and Homa Javahery Concordia University, Department of Computer Science, Canada Copyright  2004 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777 Email (for orders and customer service enquiries): cs-books@wiley.co.uk Visit our Home Page on www.wileyeurope.com or www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to permreq@wiley.co.uk, or faxed to (+44) 1243 770620. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging-in-Publication Data Multiple user interfaces : cross-platform applications and context-aware interfaces / edited by Ahmed Seffah & Homa Javahery. p. cm. Includes bibliographical references and index. ISBN 0-470-85444-8 1. Computer interfaces. I. Seffah, Ahmed. II. Javahery, Homa. TK7887.5.M86 2003 004.6 – dc22 2003057602 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 0-470-85444-8 Typeset in 10/12pt Times by Laserwords Private Limited, Chennai, India Printed and bound in Great Britain by TJ International, Padstow, Cornwall This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production. Contents Acknowledgements xv About the Editors xvii Contributors xix PART I BASIC TERMINOLOGY, CONCEPTS, AND CHALLENGES 1 1 Executive Summary and Book Overview 3 Ahmed Seffah and Homa Javahery 1.1 Motivation 3 1.2 A Few Definitions 4 1.3 Challenges 5 1.4 Specific Objectives 5 1.5 Audience 6 1.6 Overview 6 References 9 2 Multiple User Interfaces: Cross-Platform Applications and Context-Aware Interfaces 11 Ahmed Seffah and Homa Javahery 2.1 MUI: Characterization and Evolution 11 2.1.1 Interaction Styles 13 2.1.2 Fundamental Characteristics 15 2.1.3 Vertical versus Horizontal Usability 16 2.1.4 Related Work 16 2.2 Fertile Topics for Research Exploration 18 2.2.1 Context-Aware Development 18 2.2.2 Model-Based Development 20 vi CONTENTS 2.2.3 Pattern-Driven Development 22 2.2.4 Device-Independent Development 23 2.3 Concluding Remarks 24 Acknowledgements 25 References 25 PART II ADAPTATION AND CONTEXT-AWARE USER INTERFACES 27 3 A Reference Framework for the Development of Plastic User Interfaces 29 David Thevenin, Joe¨lle Coutaz, and Gae¨lle Calvary 3.1 Introduction 29 3.2 Terminology: Context of Use, Plastic UI and Multi-Target UI 30 3.2.1 Context of Use and Target 30 3.2.2 Multi-Target User Interfaces and Plastic User Interfaces 31 3.2.3 Terminology: Summary 32 3.3 The “Plastic UI Snowflake” 32 3.3.1 Target Sensitivity 33 3.3.2 Classes of Software Tools 33 3.3.3 Actors in Charge of Adaptation 34 3.3.4 Computation of Multi-Target and Plastic User Interfaces 35 3.3.5 User Interface Software Components 35 3.3.6 User Interface Migration 37 3.4 The Process Reference Framework for Multi-Target and Plastic UIs 37 3.4.1 General Description 38 3.4.2 The Process Reference Framework in the Design Phase 39 3.4.3 Instantiations of the Process Reference Framework 41 3.5 ARTStudio: An Application of the Process Reference Framework 43 3.5.1 The EDF Home Heating Control System 43 3.5.2 ARTStudio 43 3.6 Conclusion 49 Acknowledgement 49 References 49 4 Temporal Aspects of Multi-Platform Interaction 53 David England and Min Du 4.1 Introduction 53 4.2 Temporal Contexts of Multiple Platforms 55 4.2.1 Fitts’ Law and the Control:Display Ratio 55 4.2.2 Computation Speed of the Platform 56 4.2.3 Support for Task Switching on Platforms 56 4.3 Modelling Temporal Contexts 57 4.3.1 Action Selection Pattern 58 4.3.2 Progress Monitoring Pattern 59 CONTENTS vii 4.3.3 Task Management Pattern 61 4.3.4 Platform Interaction Pattern 62 4.4 The Temporal Constraint Engine 63 4.5 Discussion 64 4.6 Conclusions 65 References 65 A. The PUAN Notation 66 5 The PALIO Framework for Adaptive Information Services 69 Constantine Stephanidis, Alexandros Paramythis, Vasilios Zarikas, and Anthony Savidis 5.1 Introduction 69 5.2 The PALIO System Architecture 71 5.2.1 Overview 71 5.2.2 The PALIO Adaptation Infrastructure 75 5.3 PALIO as an Adaptive Hypermedia System 76 5.3.1 Adaptation Determinants 77 5.3.2 Decisions on the Basis of Adaptation Determinants 78 5.3.3 Adaptation Actions 80 5.4 PALIO in the Context of MUI 83 5.4.1 PALIO as a Web UI 83 5.4.2 A Brief Example 88 5.5 Summary and On-Going Work 89 Acknowledgements 90 References 90 Footnotes 91 PART III DEVELOPMENT TECHNOLOGY AND LANGUAGES 93 6 Building Multi-Platform User Interfaces with UIML 95 Mir Farooq Ali, Manuel A. Pe´rez-Quin˜ones, and Marc Abrams 6.1 Introduction 95 6.2 Terminology 97 6.3 Related Work 98 6.4 UIML 100 6.4.1 Language Overview 101 6.4.2 The Component 101 6.4.3 The Component 102 6.4.4 A Sample UI 102 6.5 A Framework for Multi-Platform UI Development 104 6.5.1 Task Model 105 6.5.2 Generic Description of Device Families 106 6.5.3 Abstract to Concrete Transformations 109 viii CONTENTS 6.6 Transformation-Based UI Development Environment 111 6.6.1 TIDE Version 1 112 6.6.2 Goals for TIDE 2 112 6.7 Conclusions 115 Acknowledgements 116 References 116 7 XIML: A Multiple User Interface Representation Framework for Industry 119 Angel Puerta and Jacob Eisenstein 7.1 Introduction 119 7.1.1 Special Challenges for MUI Solutions for Industry 120 7.1.2 Foundation Technologies 121 7.1.3 Summary of Chapter 121 7.2 The XIML Representation Framework 121 7.2.1 Target Computing Model 122 7.2.2 XIML Requirements 123 7.2.3 Structure and Organization of XIML 124 7.2.4 Validation Exercises 127 7.3 An XIML Pilot Application 133 7.3.1 MANNA: The Map Annotation Assistant 134 7.3.2 The MANNA Abstract XIML Components 136 7.3.3 XIML-Based Middleware for MANNA 139 7.4 Discussion 144 7.4.1 The XIML Roadmap 144 7.4.2 Related Work 145 7.4.3 Summary of Findings 146 Acknowledgements 146 References 146 8 AUIT: Adaptable User Interface Technology, with Extended Java Server Pages 149 John Grundy and Wenjing Zou 8.1 Introduction 149 8.2 Case Study: A Collaborative Job Management System 151 8.3 Related Work 152 8.4 Our Approach 154 8.5 Design and Implementation 156 8.6 Job Management System Examples 161 8.7 Experiences 164 8.8 Summary 166 References 166 CONTENTS ix PART IV MODEL-BASED DEVELOPMENT 169 9 Adaptive Task Modelling: From Formal Models to XML Representations 171 Peter Forbrig, Anke Dittmar, and Andreas Mu¨ller 9.1 Introduction 171 9.2 Model-Based Software Development 172 9.2.1 Models Used in the Design Process 172 9.2.2 Task Modelling 172 9.2.3 New Challenges for Modelling 175 9.3 Adaptive Specification Techniques 176 9.3.1 Adapted Task Models 177 9.3.2 Specification of Device Features by XML 179 9.4 Example of an Electronic Shop 181 9.4.1 The Task Model of E-Shopping 181 9.4.2 The Generation of Specific User Interfaces 183 9.5 Conclusions 191 References 191 10 Multi-Model and Multi-Level Development of User Interfaces 193 Jean Vanderdonckt, Elizabeth Furtado, Joa˜o Jose´ Vasco Furtado, Quentin Limbourg, Wilker Bezerra Silva, Daniel William Tavares Rodrigues, and Leandro da Silva Taddeo 10.1 Introduction 193 10.2 Related Work 194 10.3 Definition of Model 195 10.4 Conceptual Level 198 10.4.1 Definition 198 10.4.2 Case Study 199 10.5 Logical Level 201 10.5.1 Definition 201 10.5.2 Case Study 202 10.6 Physical Level 205 10.6.1 Definition 205 10.6.2 Case Study 205 10.7 Summary of the Development Process 210 10.8 Conclusion 213 Acknowledgements 214 References 215 11 Supporting Interactions with Multiple Platforms Through User and Task Models 217 Luisa Marucci, Fabio Paterno`, and Carmen Santoro 11.1 Introduction 217 x CONTENTS 11.2 An Illustrative Scenario 219 11.3 General Description of the Approach 221 11.4 Role of the Task Model in Design 223 11.4.1 From the Task Model to the Abstract User Interface 225 11.4.2 The Language for Abstract User Interfaces 226 11.4.3 From the Abstract User Interface to its Implementation 228 11.5 Relations between Task Model and User Model 228 11.6 The User Model 229 11.7 Adaptive Rules 232 11.7.1 Navigation as a Function of Task Frequency 232 11.7.2 Navigation as a Function of Task Performance 234 11.7.3 Modification of Presentation 235 11.7.4 Modification of Content Presentation 235 11.8 Conclusions 237 Acknowledgements 237 References 238 PART V ARCHITECTURES, PATTERNS, AND DEVELOPMENT TOOLKITS 239 12 Migrating User Interfaces Across Platforms Using HCI Patterns 241 Homa Javahery, Ahmed Seffah, Daniel Engelberg, and Daniel Sinnig 12.1 Introduction 241 12.2 A Brief Overview of HCI Patterns 243 12.3 Redesigning User Interfaces with Pattern Mapping 245 12.3.1 The Effect of Screen Size on Redesign 245 12.3.2 Pattern-based Redesign: A Case Study with Navigation Patterns 247 12.3.3 Architecture Size as an Added Variable in Redesign 248 12.4 Research Directions for the Use of Patterns in Reengineering User Interfaces 254 12.4.1 Pattern-Assisted Reengineering 255 12.4.2 Comparing Reengineering to Redesign 256 12.5 Conclusion and Future Investigations 257 Acknowledgements 259 References 259 13 Support for the Adapting Applications and Interfaces to Context 261 Anind K. Dey and Gregory D. Abowd 13.1 Introduction 261 13.2 Why Context is Difficult to Use and Why Support is Needed for it 264 13.2.1 Separation of Concerns 264 13.2.2 Context Interpretation 265 CONTENTS xi 13.2.3 Transparent, Distributed Communications 265 13.2.4 Constant Availability of Context Acquisition 266 13.2.5 Context Storage and History 266 13.2.6 Resource Discovery 266 13.3 Basic Component-Based Architecture and the Conference Assistant Application 267 13.3.1 Context Widgets 268 13.3.2 Context Aggregators 268 13.3.3 Context Interpreters 269 13.3.4 Services 269 13.3.5 Discoverers 270 13.3.6 Conference Assistant Application 270 13.3.7 Summary 276 13.4 Situation Support and the CybreMinder Application 276 13.4.1 Implementation of the Situation Abstraction 277 13.4.2 CybreMinder: A Complex Example that Uses the Situation Abstraction 278 13.4.3 Summary 283 13.5 Fusion Support and the In/Out Board Application 284 13.5.1 The Architecture of the Location Service 285 13.5.2 Representing Location 286 13.5.3 Details on Positioning Systems 287 13.5.4 Fusion and Aggregation of Location 289 13.5.5 Accessing, Interpreting and Handling Location Data Within an Application 289 13.5.6 Sample Application Development 291 13.5.7 Summary 292 13.6 Conclusions 293 Acknowledgements 294 References 294 14 A Run-time Infrastructure to Support the Construction of Distributed, Multi-User, Multi-Device Interactive Applications 297 Simon Lock and Harry Brignull 14.1 Introduction 297 14.2 MUI Interaction Scenario 298 14.3 Requirements for Infrastructure 299 14.4 Existing Approaches 301 14.5 Design of Infrastructure and Development Framework 303 14.5.1 Design of Interaction Metaphor 305 14.5.2 Bubble Glosses 307 14.6 Implementation of Infrastructure and Development Framework 310 14.7 Operation of the Infrastructure 311 14.7.1 Dynamic Device Service Registration 311 xii CONTENTS 14.7.2 Dynamic Device Service Selection 311 14.7.3 Application Service Linkage 312 14.7.4 Bubble Synchronisation 313 14.8 Infrastructure Utilisation 314 14.9 Application Usage Scenarios 316 14.10 Discussion 320 14.11 Conclusions 321 References 322 PART VI EVALUATION AND SOCIAL IMPACTS 325 15 Assessing Usability across Multiple User Interfaces 327 Gustav ¨Oquist, Mikael Goldstein and Didier Chincholle 15.1 Introduction 327 15.2 Multiple User Interfaces: Multiple Contexts of Use 328 15.3 Multiple Contexts of Use: Multiple Factors of Usability 330 15.3.1 Portability 330 15.3.2 Attentiveness 331 15.3.3 Manageability 333 15.3.4 Learnability 334 15.3.5 Indexical Factors of Usability for Different Contexts of Use 335 15.4 Assessing Usability of Mobile Interfaces 336 15.4.1 Mobile Input Interfaces 337 15.4.2 Mobile Output Interfaces 341 15.5 Discussion 346 15.6 Conclusions 347 References 348 16 Iterative Design and Evaluation of Multiple Interfaces for a Complex Commercial Word Processor 351 Joanna McGrenere 16.1 Introduction 351 16.2 Design Solutions to Complex Software 353 16.3 Study One 355 16.3.1 Methodology 355 16.3.2 Selected Results 356 16.4 Pilot Study 359 16.4.1 Implementation 360 16.4.2 Objectives and Methodology 361 16.4.3 Selected Results 362 CONTENTS xiii 16.5 Study Two 363 16.5.1 Methodology 365 16.5.2 Selected Results 366 16.6 Summary and Conclusions 369 Acknowledgements 371 References 371 Footnotes 372 17 Inter-Usability of Multi-Device Systems – A Conceptual Framework 373 Charles Denis and Laurent Karsenty 17.1 Introduction 373 17.2 Inter-Usability: A Conceptual Framework 374 17.2.1 Principal Processes Involved in Transitions between Devices 374 17.2.2 Requirements for Knowledge Continuity 376 17.2.3 Requirements for Task Continuity 379 17.3 Design Principles for Inter-Usability 380 17.3.1 Inter-Device Consistency 381 17.3.2 Transparency 382 17.3.3 Adaptability 383 17.4 Conclusion 384 Acknowledgements 384 References 384 Subject Index 387 Acknowledgements The help of many people made this book possible, and we are grateful to all of them. We thank our editor Birgit Gruber, at John Wiley & Sons Ltd., who guided us throughout this project. Daniel Engelberg and Jonathan Benn were indispensable for the editing process, and we thank them for their help in editing various chapters. Daniel Sinnig patiently helped with revising various chapters. Rozita Naghshin, our digital art expert, was a great source of help for advice on image layout and creation. To all the members of the HCSE (Human- Centered Software Engineering) Group, we thank you for participating in the discussion and brainstorming of this project. We thank FCAR (Le Fonds que´be´cois de la recherche sur la nature et les technologies), NSERC (National Sciences and Engineering Council of Canada), and the Faculty of Engineering, Concordia Research Chair programs, for their financial support. We are grateful to all the reviewers of this book. We were lucky enough to have a wide spectrum of international reviewers, who patiently reviewed all chapters and gave us crucial feedback. We thank John Grundy from the University of Auckland, who gave us sound advice and feedback for a number of chapters. Above all, we thank the contributors of this book. Without them, this book would not have been possible. We thank them for patiently modifying chapters, rewriting passages, and putting up with our requests. We acknowledge all of them for their efforts in making this book a success. Ahmed Seffah Homa Javahery About the Editors Ahmed Seffah is a professor in the department of Computer Science at Concordia Uni- versity. He is director of the Human-Centered Software Engineering Group and the co-founder of the Concordia Software Usability and Empirical Studies Lab. He holds a PhD in software engineering from the Ecole Centrale de Lyon (France). His research interests are at the crossroads between software engineering and Human-Computer Inter- action (HCI), including usability measurement, user interface design, empirical studies on developer experiences with CASE tools, human-centered software engineering, and patterns as a vehicle for integrating HCI knowledge in software engineering practices. Dr. Seffah is the vice-chair of the IFIP working group on user-centered design method- ologies. During the last 10 years, he has been involved in different projects in North America and Europe. Homa Javahery is a researcher and project manager with the Human-Centered Soft- ware Engineering Group, including the Usability and Empirical Studies Lab, in the department of Computer Science at Concordia University. She holds a Master’s degree in Computer Science from Concordia University, and a Bachelor of Science degree from McGill University. She is combining different design approaches from human sciences and engineering disciplines to develop a pattern-oriented framework for designing a large variety of interfaces. She has been involved in different collaborative projects at the INRIA Research Institute in Nancy, France and the Daimler-Chrysler Research Institute in Ulm, Germany. Contributors Gregory D. Abowd College of Computing Georgia Institute of Technology Atlanta, Georgia 30332-0280 USA abowd@cc.gatech.edu Marc Abrams Harmonia, Inc. PO Box 11282 Blacksburg, VA 24062 USA marc@harmonia.com Harry Brignull University of Sussex Room 5A3, Interact Lab School of Cognitive and Computing Sciences Falmer, Brighton BN1 9QH UK harrybr@cogs.susx.ac.uk +44 (0) 1273 877221 Gae¨lle Calvary IIHM Group, CLIPS-IMAG Lab BP 53, 385 rue de la Bibliotheque 38041 Grenoble Cedex 9 France Joelle.Coutaz@imag.fr +33 4 76 51 48 54 Didier Chincholle Ericsson Research ERA/TVU/U Torshamnsgatan 23, 164 80 Kista Sweden didier.chincholle@era.ericsson.se +46 8 585 303 76 Joe¨lle Coutaz IIHM Group, CLIPS-IMAG Lab BP 53, 385 rue de la Bibliotheque 38041 Grenoble Cedex 9 France Joelle.Coutaz@imag.fr +33 4 76 51 48 54 Charles Denis INTUILAB Prologue 1, La Pyre´ne´enne, BP 27/01, 31312 Labe`ge Cedex France denis@intuilab.com Anind K. Dey Senior Researcher, Intel Research 2150 Shattuck Ave, Suite 1300 Berkeley, CA 94704 USA anind@intel-research.net +1-510-495-3012 xx CONTRIBUTORS Anke Dittmar University of Rostock Department of Computer Science Albert-Einstein-Str. 21 D-18051 Rostock Germany ad@informatik.uni-rostock.de Min Du Liverpool John Moores University School of Computing and Mathematical Sciences Byrom St, Liverpool L3 3AF UK edcmdu@livjm.ac.uk +44 (0) 151 231 2271 Jacob Eisenstein CEO – RedWhale Software 277 Town & Country Village Palo Alto CA 94303 USA jacobe@mit.edu +1 650 321-3425 Daniel Engelberg CGI Group Inc. 1130 Sherbrooke West, 7th floor Montreal, Quebec H3A 2M8 Canada dan.engelberg@sympatico.ca +1 514-281-7000, local 5820 David England Liverpool John Moores University School of Computing and Mathematical Sciences Byrom St, Liverpool L3 3AF UK d.england@livjm.ac.uk +44 (0) 151 231 2271 Mir Farooq Ali Virginia Technology Institute Department of Computer Science (0106) 660 McBryde Hall Blacksburg, VA 24061 USA mfali@cs.vt.edu 1(540) 231 1927 Peter Forbrig University of Rostock Department of Computer Science Albert-Einstein-Str. 21 D-18051 Rostock Germany pforbrig@informatik.uni-rostock.de Elizabeth Furtado Universidade de Fortaleza NATI – Ce´lula EAD Washington Soares, 1321 Bairo Edson Queiroz Fortaleza (Ceara´), BR-60455770 Brazil elizabet@unifor.br Joa˜o Jose´ Vasco Furtado Universidade de Fortaleza NATI – Ce´lula EAD Washington Soares, 1321 Bairo Edson Queiroz Fortaleza (Ceara´), BR-60455770 Brazil vasco@unifor.br Mikael Goldstein Ericsson Research ERA/TVU/U Torshamnsgatan 23 164 80 Kista Sweden mikael.goldstein@era.ericsson.se +46 8 757 3679 John Grundy University of Auckland Department of Computer Science Private Bag 92019 Auckland CONTRIBUTORS xxi New Zealand john-g@cs.auckland.ac.nz +64-9-3737-599 ext. 8761 Homa Javahery Department of Computer Science Faculty of Engineering and Computer Science 1455 de Maisonneuve Blvd West Montreal, Quebec H3G 1M8 Canada h javahe@cs.concordia.ca +1 514-848-3024 Laurent Karsenty INTUILAB Prologue 1, La Pyre´ne´enne, BP 27/01, 31312 Labe`ge Cedex France karsenty@intuilab.com Quentin Limbourg Universite´ catholique de Louvain (UCL) Information System Unit (ISYS-BCHI) Institut d’Administration et de Gestion (IAG) Place des Doyens, 1 B-1348 Louvain-la-Neuve Belgium limbourg@isys.ucl.ac.be +32-10.47.85.25 Simon Lock Lancaster University Computing Department Lancaster LA1 4YR UK lock@comp.lancs.ac.uk +44-1524-592795 Luisa Marucci ISTI-CNR Via G. Moruzzi 1 56100 Pisa Italy luisa.marucci@guest.cnuce.cnr.it +39 050 3153066 Joanna McGrenere University of British Columbia Department of Computer Science 201–2366 Main Mall Vancouver, BC V6J 2E2 Canada joanna@cs.ubc.ca 604-827-5201 Andreas Mu¨ller University of Rostock Department of Computer Science Albert-Einstein-Str. 21 D-18051 Rostock Germany Xray@informatik.uni-rostock.de Gustav ¨Oquist Bollhusgra¨nd 7 113 31 Stockholm Sweden gustav@stp.ling.uu.se +46 8 739 417 783 Alexandros Paramythis Foundation for Research and Technology – Hellas Institute of Computer Science Science and Technology Park of Crete Heraklion, Crete GR – 71110 Greece cs@ics.forth.gr +30-810-391741 Fabio Paterno` ISTI-CNR Via G. Moruzzi 1 56100 Pisa Italy fabio.paterno@cnuce.cnr.it +39 050 3153066 xxii CONTRIBUTORS Manuel Pe´rez-Quin˜ones Virginia Technology Institute Department of Computer Science (0106) 660 McBryde Hall Blacksburg, VA 24061 USA perez@cs.vt.edu 1(540) 231 2646 Angel R. Puerta CEO – RedWhale Software 277 Town & Country Village Palo Alto CA 94303 USA puerta@redwhale.com +1 650 321-3425 Carmen Santoro ISTI-CNR Via G. Moruzzi 1 56100 Pisa Italy C.Santoro@cnuce.cnr.it +39 050 3153066 Anthony Savidis Foundation for Research and Technology – Hellas Institute of Computer Science Science and Technology Park of Crete Heraklion, Crete, GR – 71110 Greece cs@ics.forth.gr +30-810-391741 Ahmed Seffah Concordia University Department of Computer Science Faculty of Engineering and Computer Science 1455 de Maisonneuve Blvd West Montreal, Quebec H3G 1M8 Canada seffah@cs.concordia.ca +1 514-848-3024 Wilker Bezerra Silva Universidade de Fortaleza NATI – Ce´lula EAD Washington Soares, 1321 Bairo Edson Queiroz Fortaleza (Ceara´), BR-60455770 Brazil wilker@unifor.br Daniel Sinnig Concordia University Department of Computer Science Faculty of Engineering and Computer Science 1455 de Maisonneuve Blvd West Montreal, Quebec H3G 1M8 Canada +1 514-848-3024 Constantine Stephanidis Foundation for Research and Technology – Hellas Institute of Computer Science Science and Technology Park of Crete Heraklion, Crete, GR – 71110 Greece cs@ics.forth.gr +30-810-391741 Leandro da Silva Taddeo Universidade de Fortaleza NATI – Ce´lula EAD Washington Soares, 1321 Bairo Edson Queiroz Fortaleza (Ceara´), BR-60455770 Brazil taddeo@unifor.br Daniel William Tavares Rodrigues Universidade de Fortaleza NATI – Ce´lula EAD Washington Soares, 1321 Bairo Edson Queiroz Fortaleza (Ceara´), BR-60455770 CONTRIBUTORS xxiii Brazil danielw@unifor.br David Thevenin IIHM Group, CLIPS-IMAG Lab BP 53, 385 rue de la Bibliotheque 38041 Grenoble Cedex 9 France Joelle.Coutaz@imag.fr +33 4 76 51 48 54 Jean Vanderdonckt Universite´ catholique de Louvain (UCL) Information System Unit (ISYS-BCHI) Institut d’Administration et de Gestion (IAG) Place des Doyens, 1 B-1348 Louvain-la-Neuve Belgium vanderdonckt@isys.ucl.ac.be +32-10.47.85.25 Vasilios Zarikas Foundation for Research and Technology – Hellas Institute of Computer Science Science and Technology Park of Crete Heraklion, Crete, GR – 71110 Greece cs@ics.forth.gr +30-810-391741 Wenjing Zou University of Auckland Department of Computer Science Private Bag 92019 Auckland New Zealand wenjingzou@hotmail.com +64-9-3737-599 ext. 8761 Part I Basic Terminology, Concepts, and Challenges 1Executive Summary and Book Overview Ahmed Seffah and Homa Javahery Human-Centered Software Engineering Group, Department of Computer Science, Concordia University, Canada 1.1. MOTIVATION In recent years, a wide variety of computer devices including mobile telephones, personal digital assistants (PDAs) and pocket PCs has emerged. Many existing devices are now being introduced as an alternative to traditional computers. Internet-enabled television (WebTV), 3D-interactive platforms with voice capabilities, and electronic whiteboards attached to desktop machines are among the many examples. In addition, we are moving away from the dominance of the WIMP (Windows, Icons, Mouse, and Pointer) system as a main metaphor of human-computer interaction. Novel interaction styles are emerging. These include web applications where users interact with the content, interactive television controlled by hand-held remotes, and PDAs with small screens and styli for gesture-based interaction. All these variations in devices and interaction styles require changes in design, devel- opment and testing frameworks. This book aims to introduce the reader to the current research trends and innovative frameworks being developed to address these changes. Multiple User Interfaces. Edited by A. Seffah and H. Javahery  2004 John Wiley & Sons, Ltd ISBN: 0-470-85444-8 4 AHMED SEFFAH AND HOMA JAVAHERY 1.2. A FEW DEFINITIONS This book refers to several context-specific terms including: • Multi-device user interfaces: These allow a user to interact using various kinds of computers including traditional office desktop, laptop, palmtop, PDA with or without keyboards, and mobile telephone. • Cross-platform user interfaces: These can run on several operating systems including Windows, Linux and Solaris, if the user interface (UI) code is portable. For example, Java runs a virtual machine called JVM, and code is compiled into an intermediate format known as Java byte code, which is platform independent. When Java byte code is executed within the JVM, the JVM optimizes the code for the particular platform on which it is running. Microsoft’s latest technology,. NET follows the same principles. Code is compiled into Microsoft Intermediate Language (MSIL) and is then executed within the. NET framework as an application domain. • Mobile versus stationary/fixed user interfaces: A mobile platform gives users seamless access to information and services even when they are moving. Mobile computing includes a large variety of mobile phones and PDAs, as well as new devices such as wireless MP3 music players, digital cameras and personal health monitors. • Context-aware applications: These refer to the ability of computing devices to detect, sense, interpret and respond to aspects of a user’s local environment and the computing devices themselves. • User interface plasticity: The term plasticity is inspired from the property of mate- rials that expand and contract under natural constraints without breaking, thus preserving continuous use. Applied to HCI, plasticity is the capacity of an inter- active system to withstand variations of contexts of use while preserving usability properties. • Universal user interfaces: These can support a broad range of hardware, software and network capabilities with the central premise of accommodating users with a variety of characteristics. These characteristics include diversity in skills, knowledge, age, gender, disabilities, disabling conditions (mobility, sunlight, noise), literacy levels, cultures, income levels, etc. [Hochheiser and Shneiderman 2001]. • Multiple user interfaces (MUI): These provide different views of the same information and coordinate the services available to users from different computing platforms. By computing platform, we refer to a combination of hardware, computing capabilities, operating system and UI toolkit. The hardware includes traditional office desktops, laptops, palmtops, mobile telephones, personal digital assistants (PDAs) and interac- tive television. In a larger sense, computing platforms include wearable computers and any other real or virtual objects that can interact with the services and informa- tion. MUIs can support different types of look-and-feel and offer different interaction styles. These different types of look-and-feel and interaction styles should take into account the constraints of each computing platform while maintaining cross-platform consistency. EXECUTIVE SUMMARY AND BOOK OVERVIEW 5 1.3. CHALLENGES Olsen et al. [2000], Johnson [1998] and Brewster et al. [1998] highlight the design chal- lenges associated with the small screen size of hand-held devices. In comparison to desktop computers, hand-held devices always suffer from a lack of screen real estate. Therefore, new interaction metaphors have to be invented for such devices. Many assumptions about classical stationary applications no longer apply for hand- held devices due to the wide range of possibilities currently available. This wide range of possibilities is due to hand-held devices having constantly updated capabilities, exploiting additional features of novel generations of networks, and often being enabled for mobile users with varying profiles. Furthermore, many web-based UIs adapt to client devices, users and user tasks [see Chapters 8 and 10]. This adaptation provides interfaces that run on conventional web browsers using HyperText Markup Language (HTML), as well as on wireless PDAs, mobile phones and pagers using Wireless Markup Language (WML) [see Chapter 5]. In addition, it is important to adapt UIs to different users and user tasks [see Chapters 7 and 8]. For example, it is necessary to hide “Update” and “Delete” buttons if the user is a customer or if the user is a staff member performing only information retrieval tasks. Building such interfaces using current web-based system implementation technologies is difficult and time-consuming, resulting in hard-to-maintain solutions. Universal design is emerging as an approach where user interfaces of an interactive application have to be designed for the widest population of users in different contexts of use. In particular, the multiplicity of parameters dramatically increases the complexity of the design phase by adding many design options from which to choose. In addition, methods for developing UIs do not mesh well with this variety of parameters as they are not identified and manipulated in a structured way, nor truly considered in the design process [see Chapter 10]. 1.4. SPECIFIC OBJECTIVES Even if the software tools for developing a large variety of interfaces on each computing platform are already available or will be in the near future [Myers 2000], the following are the major development issues that need to be addressed by both academic and industrial researchers: • Building the ability to dynamically respond to changes in the environment such as network connectivity, user’s location, ambient sound and lighting conditions: How can we adapt the UI to the diversity of computing platforms that exists today? How can we maintain or adapt the high level of interactivity of the traditional office desktop in small devices without a keyboard and mouse? How can we make it possible for users to customize a device? When a single device is customized, how can this customization be reflected on all of the other devices available to the user? 6 AHMED SEFFAH AND HOMA JAVAHERY • Designing for universal usability: What kinds of design methods are needed for design- ing for diverse users and a large variety of technologies? Are the design techniques for UI modelling suitable for addressing the problems of diversity, cross-platform consis- tency and universal accessibility? • Checking consistency between versions for guaranteeing seamless interaction across multiple devices: Should we strive for uniformity in the services offered, dialogue styles and presentation formats, or should we adapt the interfaces to the constraints and capabilities of each device and/or each context of use? When designing MUIs, what is the best way to take into account the constraints related to each type of device while ensuring maintainability and cross-platform consistency of interfaces? • Implementing and maintaining versions of the user interface across multiple devices: How can we implement and validate a MUI for d devices without writing p programs, training an army of developers in l languages and UI toolkits, and maintaining l*p architectural models for describing the same UI? Are the content markup languages adequate for device-independent authoring? The book also introduces a variety of development frameworks that have been investigated over the last few years: • Conceptual and adaptation frameworks for interacting with multiple user interfaces, including visual and awareness metaphors, and specific interaction styles; • Design frameworks and patterns including widgets, toolkits and tools for multi-device development and in particular for mobile devices; • Application frameworks that use multi-devices or multiple user interfaces, in particu- lar collaborative work environments, distance education systems and remote software deployment systems; • Validation frameworks including usability techniques for testing multiple user inter- faces, as well as empirical tests and feedback. 1.5. AUDIENCE This book introduces design and development frameworks for multi-device, context-aware and multiple user interface systems. These frameworks are valuable to researchers and practitioners in usability and software engineering, and generally to anyone interested in the problem of developing and validating multi-devices or cross-platform user interfaces. User interface developers, students and educators can use these frameworks to extend and improve their HCI methodologies, and to learn techniques for developing and evaluating a multiple user interface. 1.6. OVERVIEW This book is divided into 6 parts: Part I discusses “Basic Terminology, Concepts, and Challenges”. Following the execu- tive summary, in Chapter 2 Ahmed Seffah and Homa Javahery, the co-editors of this book, EXECUTIVE SUMMARY AND BOOK OVERVIEW 7 present a broad overview of multiple user interfaces. They discuss promising development models that can facilitate MUI development while increasing cross-platform usability. This chapter is highly speculative and will provide questions for basic research. This is a selec- tive list of topics, and not exhaustive. The goal is to give researchers and practitioners a glimpse of the most important problems surrounding MUI design and development. The editors’ opinions expressed in Chapter 2 do not necessarily reflect all of the contributors’ ideas. Complementary and differing opinions are presented by other contributors in their own chapters. After exploring these first two chapters, the reader should have an increased awareness of the diversity of computing platforms and devices, a deeper understanding of the major development challenges and difficulties, and a familiarity with the basic terminology used. Part II is entitled “Adaptation and Context-Aware User Interfaces”, and provides three traditional but comprehensive perspectives on adaptation and context-aware techniques. David Thevenin et al. from the CLIPS-IMAG Laboratory in Grenoble, France, introduce the novel concept of user interface plasticity in Chapter 3. This chapter also provides a generous glossary of terms complementing the basic terminology presented in Chapter 2. Chapters 4 and 5 examine two dimensions of multi-interaction and adaptation. David England and Min Du, from Liverpool John Moores University, take a look at the dif- ferent temporal characteristics of platforms that can affect user performance. They then propose a framework taking into account temporal aspects of interaction in the use of different devices. They describe how the temporal aspects should be incorporated into the interaction design process. In Chapter 5, Constantine Stephanadis et al., from the Institute of Computer Science of the Foundation for Research and Technology – Hellas, Greece, introduce their framework called PALIO (Personalized Access to Local Information and services for tourists), focusing on its extensive support for adaptation. They demonstrate how PALIO has been successfully used in the development of a real-world context-aware information system for tourists using a wide range of devices. Part III is on “Development Technology and Languages” and consists of three differ- ent XML-based development frameworks. In Chapter 6, Mir Farooq Ali et al. (from the Virginia Technology Institute and Harmonia Inc.) describe a high level XML-based User Interface Markup Language (UIML) for the development of cross-platform user interfaces. In Chapter 7, Angel Puerta and Jacob Eisenstein, from RedWhale Software, discuss the rationale of XIML, another XML-based language for developing multiple user interfaces by transforming and refining tasks and UI models. These modelling and programming languages distinguish the concrete aspects of a user interface such as presentation and dialogue from its abstract aspects including the context and the tasks. They are consid- ered by the research community to be a bridge across the gap between the design and development of user interfaces. John Grundy and Wenjing Zou, from the University of Auckland in New Zealand, go a step further by showing how a UI markup language can be interfaced with existing programming languages. They describe how scripts written in their device-independent markup language (AUIT) can be embedded in conventional Java server pages to provide a single adaptable thin-client interface for web-based systems. At run-time, AUIT uses a single interface description to automatically provide an interface for multiple web devices such as desktop HTML and mobile WML-based systems, while highlighting, hiding or disabling the interface elements depending on the current context. 8 AHMED SEFFAH AND HOMA JAVAHERY Together, these three chapters show how XML-based markup languages, with the help of model-based techniques, can lead to an advanced framework for the development of multi-platform user interfaces. Part IV, on “Model-Based Development”, includes three chapters describing the state of the art and the needed evolution in model-based development approaches. The basic purpose of model-based approaches is to identify useful abstractions highlighting the main UI aspects that should be considered when designing effective interactive appli- cations. In Chapter 9, Peter Forbrig et al. from the University of Rostock, Germany, present two techniques for task modelling and specification. The first technique allows separate descriptions of general temporal relations within a task model versus tempo- ral constraints that are imposed by the target platform. The second technique helps to distinguish between an abstract interaction and specific representations. Using these two techniques, specific features of devices are specified by XML descriptions. In Chapter 10, Vanderdonckt et al. (from the Universite´ Catholique de Louvain, Belgium and Universi- dade de Fortaleza, Brazil) use several models at different levels in their methodological framework for universal design. First, the design process is instantiated at a conceptual level where a domain expert defines an ontology of concepts, relationships and attributes of the domain of discourse, including user modelling. Then at a logical level, a designer specifies multiple models based on the previously defined ontology and its allowed rules. The last step consists of using a physical level to develop multiple user interfaces from the previously specified models, with design alternatives determined by characteristics in the user models. Fabio Paterno`, the father of CTT (ConcurTaskTrees) notation, and his colleagues at ISTI-CNR, Italy, explain in Chapter 11 how the user model can be struc- tured for a MUI. In particular, they show how information on user preferences and on the mobile versus stationary environment (such as location and surroundings) can be used to adapt a user interface at run-time and at design time. Part V is dedicated to “Architectures, Patterns and Development Toolkits”. Homa Javahery et al. from the Human-Centered Software Engineering Group at Concordia University, discuss in Chapter 12 the role of HCI patterns and software reengineering techniques in migrating traditional GUIs to web and mobile user interfaces. In Chapter 13, Anind Dey, from Intel Research in California, and Gregory D. Abowd, from the Geor- gia Institute of Technology, present the Context Toolkit, an innovative and integrative infrastructure for the development of context-aware applications. Through the description of a number of built applications, they discuss a low-level widget abstraction that mir- rors the use of graphical widgets for building graphical user interfaces and a situation abstraction that supports easier and higher-level application development. In Chapter 14, Simon Lock from Lancaster University and Harry Brignull from Sussex University, UK, describe a run-time infrastructure including a developer-level framework. This infrastruc- ture supports the construction of applications that allow multiple users to interact through a dynamic set of interaction devices. “Evaluation and Social Impacts” are addressed in Part VI. In Chapter 15, Gustav ¨Oquist et al. (from Uppsala University in Sweden and Ericsson Research) discuss “Assessing Usability Across Multiple User Interfaces”. They present their practical experiences with stationary versus mobile usability evaluations. In particular, they outline four typical con- texts of use that they characterized by monitoring four environmental usability factors. By EXECUTIVE SUMMARY AND BOOK OVERVIEW 9 assessing these factors, it was possible to obtain a profile of how useful a given interface can be in a certain context of use. The usability profiles for several different interfaces for input and output are also presented in this chapter, partly to illustrate how usability can be assessed over multiple user interfaces, and partly as an illustration of how differ- ent interfaces have been adapted to mobile environment attributes. In Chapter 16, Joanna McGrenere from the University of British Columbia summarizes her MUI evaluation experiment combining three usability studies. In the first study, McGrenere conducted a broad-based assessment of 53 users of MS-Word 97. Based on the findings from this study, she developed a first multiple-interface prototype for MS-Word 2000 including one personalizable interface. In the last study, personalization was achieved through the Wizard of Oz technique. Even if McGrenere’s definition of a multiple user interface is restrictive compared to the ones proposed in this book, her empirical results are very promising. They demonstrate how users were better able to navigate through the menus and toolbars and learn a multiple-interface prototype. Continuing part VI, Charles Denis and Laurent Karsenty from IntuiLab Inc., argue in chapter 17 that the usability of individual devices is not sufficient: a multi-device system needs to also be inter-usable. They define inter-usability as the ease with which users transfer what they have learned from previous uses of a service when they access the service on a new device. Based on theoretical considerations and empirical observations gathered from a study with France Telecom, they propose an analysis grid combining two types of continuity, namely knowledge and task, with ergonomic design principles includ- ing consistency, transparency, and dialogue adaptability. The concept of inter-usability is very similar to the concept of horizontal usability introduced in Chapter 2 by the editors. Inter-usability or horizontal usability is a new dimension for studying the usability of MUIs and multi-device user interfaces. REFERENCES Brewster, S., Leplaˆtre, G. and Crease, M. (1998) Using Non-Speech Sounds in Mobile Comput- ing Devices. Proceedings of the First Workshop on Human Computer Interaction with Mobile Devices, May 21–23, 1998, Glasgow, UK. Hochheiser, H. and Shneiderman, B. (2001) Universal Usability Statements: Marking the trail for all users. ACM Interactions 8(2), March–April 2001, 16–18. Johnson, P. (1998) Usability and Mobility: Interactions on the move. Proceedings of the First Work- shop on Human Computer Interaction With Mobile Devices, May 21–23, 1998, Glasgow, UK. Myers, B., Hudson, S. and Pausch, R. (2000) Past, Present, and Future of User Interface Software Tools. ACM Transactions on Computer-Human Interaction, 7, 3–28. Olsen, D., Jefferies, S., Nielsen, T. et al. (2000) Cross-modal Interaction using XWeb. Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, UIST’2000, November 5–8, 2000, San Diego, USA. 2Multiple User Interfaces: Cross-Platform Applications and Context-Aware Interfaces Ahmed Seffah and Homa Javahery Human-Centered Software Engineering Group, Department of Computer Science, Concordia University, Canada 2.1. MUI: CHARACTERIZATION AND EVOLUTION We introduced the concept of “Multiple User Interface” (MUI) at the IHM-HCI 2001 workshop [Seffah et al. 2001]. Others are also using the term MUI with varying definitions [McGrenere et al. 2002; Vanderdonckt and Oger 2001]. For the purposes of this book, a Multiple User Interface is defined as an interactive system that provides: • access to information and services using different computing platforms; • multiple views of the same information on these different platforms; • coordination of the services provided to a single user or a group of users. Each view should take into account the specific capabilities and constraints of the device while maintaining cross-platform consistency and universal usability. By comput- ing platform, we refer to a combination of computer hardware, an operating system and Multiple User Interfaces. Edited by A. Seffah and H. Javahery  2004 John Wiley & Sons, Ltd ISBN: 0-470-85444-8 12 AHMED SEFFAH AND HOMA JAVAHERY a user interface (UI) toolkit. Different kinds of computing platforms include traditional office desktops, laptops, palmtops, mobile telephones, personal digital assistants (PDAs), and interactive television. A MUI provides multiple views of the same information on these different platforms and coordinates the services provided to a single user or a group of users. Each view should take into account the specific capabilities and constraints of the device while maintaining cross-platform consistency and universal usability. The informa- tion and services can reside on a single server or computer, or can be distributed among independent and heterogeneous systems. The desired views are made available on differ- ent computing platforms via the traditional client/server protocol or a direct peer-to-peer access. The concept of MUIs is highly promising in a variety of fields, such as cooperative engineering, e-commerce, on-site equipment maintenance, remote software deployment, contingency management and assistance, as well as distance education and telemedicine. As an example of MUI use, a civil engineer can use a Palm Pilot on PalmOS for gathering data when inspecting a new building. He/She can then use a mobile telephone to add comments, fax, or upload information to the office headquarters. Finally, the same engineer or any other employee can use an office workstation under Windows/Linux to analyze the data and prepare a final report. During this workflow, the engineer inter- acts with the same information and services using different variations of the UI. These variations can support differences in look-and-feel, and to a certain extent, differences in interaction style. The following is a scenario that further clarifies the MUI concept and its use, based on [Ghani 2001]: You are riding in a car with your colleague who is driving. Suddenly, your mobile phone comes on, asking if you can take a video conference call from your team in Canada to discuss a project on which you are working. You take the cal