Dr Matthias Rolf

Intrinsic motivation is a common method to facilitate exploration in reinforcement learning agents. Curiosity is thereby supposed to aid the learning of a primary goal. However, indulging in curiosity may also stand in conflict with more urgent or essential objectives such as self-sustenance. This paper addresses the problem of balancing curiosity, and correctly prioritising other needs in a reinforcement learning context. We demonstrate the use of the multi-objective reinforcement learning framework C-MORE to integrate curiosity, and compare results to a standard linear reinforcement learning integration. Results clearly demonstrate that curiosity can be modelled with the priority-objective reinforcement learning paradigm. In particular , C-MORE is found to explore robustly while maintaining self-sustenance objectives, whereas the linear approach is found to over-explore and take unnecessary risks. The findings demonstrate a significant weakness of the common linear integration method for intrinsic motivation, and the need to acknowledge the potential conflicts between curiosity and other objectives in a multi-objective framework.

Intelligent agents often have to cope with situations in which their various needs must be prioritised. Efforts have been made, in the fields of cognitive robotics and machine learning, to model need prioritization. Examples of existing frameworks include normative decision theory, the subsumption architecture and reinforcement learning. Reinforcement learning algorithms oriented towards active goal prioritization include the options framework from hierarchical reinforcement learning and the ranking approach as well as the MORE framework from multi-objective reinforcement learning. Previous approaches can be configured to make an agent function optimally in individual environments, but cannot effectively model dynamic and efficient goal selection behaviour in a generalisable framework. Here, we propose an altered version of the MORE framework that includes a threshold constant in order to guide the agent towards making economic decisions in a broad range of ‘priority-objective reinforcement learning’ (PORL) scenarios. The results of our experiments indicate that pre-existing frameworks such as the standard linear scalarization, the ranking approach and the options framework are unable to induce opportunistic objective optimisation in a diverse set of environments. In particular, they display strong dependency on the exact choice of reward values at design time. However, the modified MORE framework appears to deliver adequate performance in all cases tested. From the results of this study, we conclude that employing MORE along with integrated thresholds, can effectively simulate opportunistic objective prioritization in a wide variety of contexts.

The mechanisms of infant development are far from understood. Learning about one's own body is likely a foundation for subsequent development. Here we look specifically at the problem of how spontaneous touches to the body in early infancy may give rise to first body models and bootstrap further development such as reaching competence. Unlike visually elicited reaching, reaching to own body requires connections of the tactile and motor space only, bypassing vision. Still, the problems of high dimensionality and redundancy of the motor system persist. In this work, we present an embodied computational model on a simulated humanoid robot with artificial sensitive skin on large areas of its body. The robot should autonomously develop the capacity to reach for every tactile sensor on its body. To do this efficiently, we employ the computational framework of intrinsic motivations and variants of goal babbling-as opposed to motor babbling-that prove to make the exploration process faster and alleviate the ill-posedness of learning inverse kinematics. Based on our results, we discuss the next steps in relation to infant studies: what information will be necessary to further ground this computational model in behavioral data.

Robots that cohabitate in social spaces must abide by the same behavioural cues humans follow, including interpersonal distancing. Proxemics investigates the appropriate distances and the impact of factors affecting it, such as gender and age. This paper investigates people's attitudes towards a robot that can learn Proxemics rules by gauging direct individual feedback from a person, and utilizing it in a reinforcement learning framework. Previous learning attempts have relied on larger robots, for which physical safety is a primary concern. In contrast, our study uses a handheld sized robot that allows us to focus on the impact of distance on engageability in dialogue. General consensus between interviewees was a feeling of ease and safety during interactions, as well as disparity regarding the invasion of personal space, which was influenced by cultural background.

Both biological and artificial agents need to coordinate their behavior to suit various needs at the same time. Reconciling conflicts of different needs and contradictory interests such as self-preservation and curiosity is the central difficulty arising in the design and modelling of need and value systems. Current models of multi-objective reinforcement learning do either not provide satisfactory power to describe such conflicts, or lack the power to actually resolve them. This paper aims to promote a clear understanding of these limitations, and to overcome them with a theory-driven approach rather than ad hoc solutions. The first contribution of this paper is the development of an example that demonstrates previous approaches' limitations concisely. The second contribution is a new, non-linear objective function design, MORE, that addresses these and leads to a practical algorithm. Experiments show that standard RL methods fail to grasp the nature of the problem and ad-hoc solutions struggle to describe consistent preferences. MORE consistently learns a highly satisfactory solution that balances contradictory needs based on a consistent notion of optimality.

Robotic tele-operation systems have vast potential in areas ranging from surgical robotics and underwater exploration to disposing of toxic, explosive and nuclear materials. While visual camera feeds for the human operator are typically available and well studied, tactile sensory information is often vital for successful and efficient manipulation. Previous studies have largely focused on execution time alone as measure of success of feedback methods on individual tasks. The present study complements this by a comparative analysis of vibration and visual feedback of tactile information across a range of manipulation tasks. Results show a significant reduction in perceived workload with the implementation of vibration feedback and an improvement of error rates for visual feedback. Contrary to expectation, we did not find a reduction in task completion time. The negative finding on completion time challenges the belief that the mere existence of task-relevant feedback aids efficient task completion. The reduced workload, however, clearly points out potential for enhancing performance on more difficult and prolonged tasks with highly skilled operators.

Exploration is one of the fundamental problems in mobile robotics. Efforts to address this problem made over the past two decades divide into two approaches: reactive approaches, that make only instantaneous decisions, and map-based approaches involving e.g. grid, metric, or topological representations. Comparative studies have so far largely focused on comparing different map-based algorithms, while no common framework to compare them to purely reactive approaches currently exists. This paper aims at creating a framework to simulate, evaluate, and compare exploratory algorithms as different as reactive and map-based approaches. Preliminary results are demonstrated for two reactive algorithms, random walk and wall follower, and one map based approach, pheromone potential field, have been implemented. Measurements of navigation success, time to success, as well as computational and memory usage reveal a dominance of simple wall-following over the map-based potential field approach, and a distinct load/efficacy trade off for random walks. These preliminary results challenge the common assumptions that maps are needed for successful and efficient exploration and navigation.

AI and robot ethics have recently gained a lot of attention because adaptive machines are increasingly involved in ethically sensitive scenarios and cause incidents of public outcry. Much of the debate has been focused on achieving highest moral standards in handling ethical dilemmas on which not even humans can agree, which indicates that the wrong questions are being asked. We suggest to address this ethics debate strictly through the lens of what behavior seems socially acceptable, rather than idealistically ethical. Learning such behavior puts the debate into the very heart of developmental robotics. This paper poses a roadmap of computational and experimental questions to address the development of socially acceptable machines. We emphasize the need for social reward mechanisms and learning architectures that integrate these while reaching beyond limitations of plain reinforcement learning agents. We suggest to use the metaphor of “needs” to bridge rewards and higher level abstractions such as goals for both communication and action generation in a social context. We then suggest a series of experimental questions and possible platforms and paradigms to guide future research in the area.