ACKnowledge: A Computational Framework for Human Compatible Affordance-based Interaction Planning in Real-world Contexts

Abstract

Intelligent agents coexisting with humans often need to interact with human-shared objects in environments. Thus, agents should plan their interactions based on objects’ affordances and the current situation to achieve acceptable outcomes. How to support intelligent agents’ planning of affordance-based interactions compatible with human perception and values in real-world contexts remains under-explored. We conducted a formative study identifying the physical, intrapersonal, and interpersonal contexts that count to household human-agent interaction. We then proposed ACKnowledge, a computational framework integrating a dynamic knowledge graph, a large language model, and a vision language model for affordance-based interaction planning in dynamic human environments. In evaluations, ACKnowledge generated acceptable planning results with an understandable process. In real-world simulation tasks, ACKnowledge achieved a high execution success rate and overall acceptability, significantly enhancing usage-rights respectfulness and social appropriateness over baselines. The case study’s feedback demonstrated ACKnowledge’s negotiation and personalization capabilities toward an understandable planning process.