Most of today’s information processing systems interact with their users via built-in, pre-defined rule-based interfaces. They have no or little knowledge of the environment or their user(s). In fact these systems assume a static user environment, which can seldom be guaranteed. In the best of cases the interaction models contain some built-in knowledge of the user’s capabilities, but lack any knowledge of the current state of the user’s tacit or background knowledge, the user’s situational awareness, skills and actual “state-of-mind”. This contrasts heavily with the standard human-human interaction, which contain more elaborate patterns of use and situational awareness, thus grounding the belief that the conception and design of complex information systems that are based on cognitive capabilities will be the key differentiator between future products.  

The main focus of Dutch Companion is on improving the effectiveness of actor-agent interaction through making an agent aware of human emotions and expected social interactions. An agent can then respond not only with appropriate content but also with appropriate emotional state. The agent will have the embodiment of Philip’s iCat. The iCat is a plug & play desktop user-interface robot, capable of mechanically rendering facial expressions. The iCAT will be situated inside the kitchen. It learns to take preferences and emotional state into account when suggesting meals or activities to their inhabitants.

The project concentrates on the following:

• The ability to focus attention on the user’s needs for information and assistance as a function of  the user’s situation, goals and current (cognitive) capabilities and emotional states;
• The ability to adapt in real time to the behavior and responses beyond the mere use of some built-in static user model. 

The first point will be addressed by looking at the use of sound recognition and analysis to enhance its situational awareness. The classification of particular sounds from the environment will be used by the iCat to alert the user, depending on the current user context. Via deliberation it determines whether or not the user should be alerted.

The second point will be addressed by detection of emotion in the user’s voice (isolated from the overall sound input), combining the emotional output from other modalities, such as facial recognition and gestures. The perceived emotional state can than be used as a controlling input for the iCat’s deliberation.

The personal and social based deliberation designs and implementations will be based on agent-oriented programming, in particular 3APL. This language enables one to program the agent’s behavior based on its beliefs, goals and plans. A central element of 3APL is the way the agent deliberates how to perform plans given certain goals and beliefs, in particular which goals and plans to select and pursue, and which goals and plans to revise under the circumstances given. This so-called ‘deliberation cycle’ has to be extended to also include the effect of emotional states, as well the influence of social aspects such as norms and obligations arising from the social context. In addition we will couple the emotion recognition with the communication and dialogue iCat module, thus changing the nature of the dialogue and the content of the communication depending on the emotional state of the user. Note that interacting with agents requires integration of different fields in AI, cognitive science, computer science, etc. For example, speech recognition is still difficult in uncontrolled environments. Equally challenging still is computer vision.

If we foresee a future of actor agent communities, all these issues on the sensor/input side, as well as cognitive and actuator side need to be cross-disciplinary resolved. Starting by building a small system and deploy it in an operational setting, will uncover new unforeseen research issues. Furthermore, note that such cognitive systems will always serve specific purposes and not yet provide generic solutions to ‘universal’ problems. In different situations or context, people have different tasks, or make more or less use of certain abilities, so artificial systems should know what to support in which context. For example, a human driving a car needs different support than human inside its home.

Expected Results

The Dutch Companion project is expected to deliver concrete results based on experimentation and research regarding embodied emotional agents and natural actor-agent interaction. The expected short term results are summarized below. System Architecture From an overall actor agent system’s point of view, we foresee Dutch Companion to deliver implemented new functional system architectural insights by combining (i) sound recognition, (ii) emotion recognition, (iii) affect controlled deliberation, (iv) concept formation, (v) affect controlled communication and dialogue, and (vi) cognitive system architectures.

Validating Experiments

We develop and deploy a demonstrator framework for autonomous cognitive systems (ACSs) that show the added value of actor agent system architecture. In particular, for these purposes we assess the iCat demonstrator with its users. This assessment will be mainly conducted in the Home Lab of Philips Research in Eindhoven. The iCat will be placed in the kitchen of the Home Lab and connected to the home-network. We test how users will interact with the iCat, focusing on the effectiveness, enjoyment and acceptance of using such a domestic companion to assist and accompany a user doing (kitchen) tasks.