Dr. Sebastian Heil

Training data for user behavior models that predict subjective dimensions of visual perception are often too scarce for deep learning methods to be applicable. With the typical datasets in HCI limited to thousands or even hundreds of records, feature-based approaches are still widely used in visual analysis of graphical user interfaces (UIs). In our paper, we benchmarked the predictive accuracy of the two types of neural network (NN) models, and explored the effects of the number of features, and the dataset volume. To this end, we used two datasets that comprised over 4000 webpage screenshots, assessed by 233 subjects per the subjective dimensions of Complexity, Aesthetics and Orderliness. With the experimental data, we constructed and trained 1908 models. The feature-based NNs demonstrated 16.2%-better mean squared error (MSE) than the convolutional NNs (a modified GoogLeNet architecture); however, the CNNs’ accuracy improved with the larger dataset volume, whereas the ANNs’ did not: therefore, provided that the effect of more data on the models’ error improvement is linear, the CNNs should become superior at dataset sizes over 3000 UIs. Unexpectedly, adding more features to the NN models caused the MSE to somehow increase by 1.23%: although the difference was not significant, this confirmed the importance of careful feature engineering.

The engineering of user interfaces (UIs) increasingly relies on software tools that aid in ideation, design, evaluation, etc., but involve no real users. Particularly, user behavior models (UBMs) bear the potential to improve human-centered design processes, but their adoption in practice remains low. In this paper, we present a taxonomy for UBM tools that organizes and structures them along 7 dimensions – supported job, degree of automation, focus, interface data input, user data input, output of tool, and target interface platform. We also conduct an initial evaluation with 61 existing tools, providing insights into the current state of the field. Notably, none of the investigated tools work with user characteristics or reference interfaces as input, although this would appear very practical for real projects. Our results could support UI/UX researchers and digital design practitioners in searching for the tools and further …

Dynamic migration of code between client and server of a web application allows to balance the needs of users for smooth and responsive user interactions with the interests of software providers to reduce costs and use resources efficiently. The ability to change the execution location of parts of the application logic at runtime means that depending on client capabilities, network speed and the current load of client and server, the code distribution can be optimized. In this demonstration, we showcase dynamic code migration for a sample e-commerce web application. The demonstrator is designed according to our novel DCM architecture and uses its infrastructure to automate compilation of code fragments and manage the migration at runtime, leveraging standardized Web technologies like WebAssembly and WebSockets. Demo participants will be able to interactively control the distribution of code fragments via a …

The underlying Client/Server architecture of the Web inherently raises the question of the distribution of application logic between client and server. Currently, this distribution is static and fixed at design time, inhibiting dynamic and individual load distribution between clients and server at runtime. The benefits of dynamic migration allow balancing the needs of users, through increased responsiveness, and software providers, through better resource usage and cost reductions. Recent additions to the Web environment like WebAssembly provide a technological basis to move units of code at runtime. However, making use of them to extend a web application with dynamic code migration capabilities is challenging for web engineers. To that end, we devise a novel distributed Client/Server software architecture for web applications that supports dynamic migration of code at runtime, addressing the technical challenges …

Artificial intelligence (AI)-based user-interface (UI) design and evaluation are currently constrained by the scarcity of human-generated training data. Correspondingly, choosing appropriate neural network (NN) architecture and carefully planning the sample size is essential for building accurate machine learning models. Previously, we have estimated that for a convolutional NN to produce better mean square errors (MSEs) than feature-based models, the required training dataset size should be approximately 3000. Our current validation study with roughly 4000 web UIs and 233 subjects suggests that the estimation should be closer to 17,000. We propose corrected regression models suggesting that the dataset size effect is better described using a logarithmic function. We also report significant differences in MSEs between the employed perception dimensions, with Aesthetics models having an MSE 21.5% worse …

The main principles for designing successful UIs in a perfect world have long been known—considering many possible solutions for a problem and involving representative users in the process. In practice, however, reasons for violating those principles can be plentiful: the infamous tight budgets and schedules, lack of management buy-in, restrictions for face-to-face meetings, etc. Yet, design tools that do not require real users, such as AI-/ML-powered solutions, which could mitigate these issues seem to experience a rather low adoption rate in industry. In this paper, we present a survey with 34 professional digital designers and user researchers intended to investigate the above hypotheses. We inquire into awareness and usage of 61 such tools and platforms, as well as participants’ design and research processes and general design tool adoption in industry. From the results we identify three particular challenges …

Deep Learning techniques have become the mainstream and unquestioned standard in many fields, e.g. convolutional neural networks (CNN) for image analysis and recognition tasks. As testing and validation of graphical user interfaces (GUIs) is increasingly relying on computer vision, CNN models that predict such subjective and informal dimensions of user experience as aesthetic or complexity perception start to achieve decent accuracy. They however require huge amounts of human-labeled training data, which are costly or unavailable in the field of Human-Computer Interaction (HCI). More traditional approaches rely on manually engineered features that are extracted from UI images with domain-specific algorithms and are used in “traditional” Machine Learning models, such as feedforward artificial neural networks (ANN) that generally need fewer data. In our paper, we compare the prediction quality of …

Web Engineering becomes increasingly hungry for training data, as the application of machine learning (ML) methods in the field intensifies. Human-labeled datasets are particularly indispensable for ML-based validation and design of user interfaces (UIs). The production of such datasets is often outsourced to crowdworkers, who typically have lower motivation and payment compared to in-house staff, so the quality of their work becomes the paramount concern. In our paper, we explore the applicability of the trending fraud detection approach based on fit to power law in crowdsourced web UI labeling. On Amazon Mechanical Turk, 298 crowdworkers labeled over 30,000 UI elements in about 500 university homepage screenshots. We found a significant correlation between workers’ precisions and Kolmogorov-Smirnov statistics-based goodness-of-fit between the frequencies of UI elements in a worker’s output and …