publications

2024

1. Retrieval Augmented Generation for Natural Lanugage Query in Egocentric Videos

   Jiahao Nick Li, Li Gu, Omid Reza Heidari, and 1 more author

   In , 2024

   PDF

2. 

   OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs

   Jiahao Nick Li, Yan Xu, Tovi Grossman, and 2 more authors

   In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24), 2024

   PDF

3. 

   Human I/O: Towards a Unified Approach to Detecting Situational Impairments in Everyday Activities

   Xingyu Bruce Liu, Jiahao Nick Li, David Kim, and 2 more authors

   In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24), 2024

   PDF

2023

1. Under review

   Rocap: A Robotic Data Collection Pipeline for the Pose Estimation of Appearance-Changing Objects

   Jiahao Li, Toby Chong, Zhongyi Zhou, and 4 more authors

   2023

   PDF Video

2. In submission

   RealityPlay: Authoring Interactive and Embedded Graphics Driven by Everyday Objects with User-defined Mappings

   Jiahao Li, Ruolin Wang, Li-Yi Wei, and 2 more authors

   2023

   PDF Video

2022

1. 

   MiniKers: Interaction-Powered Smart Environment Automation

   Xiaoying Yang, Jacob Sayono, Jess Xu, and 3 more authors

   Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Sep 2022

   Abs PDF

   Automating operations of objects has made life easier and more convenient for billions of people, especially those with limited motor capabilities. On the other hand, even able-bodied users might not always be able to perform manual operations (e.g., both hands are occupied), and manual operations might be undesirable for hygiene purposes (e.g., contactless devices). As a result, automation systems like motion-triggered doors, remote-control window shades, contactless toilet lids have become increasingly popular in private and public environments. Yet, these systems are hampered by complex building wiring or short battery lifetimes, negating their positive benefits for accessibility, energy saving, healthcare, and other domains. In this paper we explore how these types of objects can be powered in perpetuity by the energy generated from a unique energy source - user interactions, specifically, the manual manipulations of objects by users who can afford them when they can afford them. Our assumption is that users’ capabilities for object operations are heterogeneous, there are desires for both manual and automatic operations in most environments, and that automatic operations are often not needed as frequently - for example, an automatic door in a public space is often manually opened many times before a need for automatic operation shows up. The energy harvested by those manual operations would be sufficient to power that one automatic operation. We instantiate this idea by upcycling common everyday objects with devices which have various mechanical designs powered by a general-purpose backbone embedded system. We call these devices, MiniKers. We built a custom driver circuit that can enable motor mechanisms to toggle between generating powers (i.e., manual operation) and actuating objects (i.e., automatic operation). We designed a wide variety of mechanical mechanisms to retrofit existing objects and evaluated our system with a 48-hour deployment study, which proves the efficacy of MiniKers as well as shedding light into this people-as-power approach as a feasible solution to address energy needed for smart environment automation.

2. 

   Roman: Making Everyday Objects Robotically Manipulable with 3D-Printable Add-on Mechanisms

   Jiahao Li, Alexis Samoylov, Jeeeun Kim, and 1 more author

   In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22), Sep 2022

   Abs PDF

   One important vision of robotics is to provide physical assistance by manipulating different everyday objects, e.g., hand tools, kitchen utensils. However, many objects designed for dexterous hand-control are not easily manipulable by a single robotic arm with a generic parallel gripper. Complementary to existing research on developing grippers and control algorithms, we present Roman, a suite of hardware design and software tool support for robotic engineers to create 3D printable mechanisms attached to everyday handheld objects, making them easier to be manipulated by conventional robotic arms. The Roman hardware comes with a versatile magnetic gripper that can snap on/off handheld objects and drive add-on mechanisms to perform tasks. Roman also provides software support to register and author control programs. To validate our approach, we designed and fabricated Roman mechanisms for 14 everyday objects/tasks presented within a design space and conducted expert interviews with robotic engineers indicating that Roman serves as a practical alternative for enabling robotic manipulation of everyday objects.

3. 

   Mobiot: Augmenting Everyday Objects into Moving IoT Devices Using 3D Printed Attachments Generated by Demonstration

   Abul Al Arabi, Jiahao Li, Xiang ’Anthony Chen, and 1 more author

   In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Sep 2022

   Abs PDF

   Recent advancements in personal fabrication have brought novices closer to a reality, where they can automate routine tasks with mobilized everyday objects. However, the overall process remains challenging- from capturing design requirements and motion planning to authoring them to creating 3D models of mechanical parts to programming electronics, as it demands expertise. We introduce Mobiot, an end-user toolkit to help non-experts capture the design and motion requirements of legacy objects by demonstration. It then automatically generates 3D printable attachments, programs to operate assembled modules, a list of off-the-shelf electronics, and assembly tutorials. The authoring feature further assists users to fine-tune as well as to reuse existing motion libraries and 3D printed mechanisms to adapt to other real-world objects with different motions. We validate Mobiot through application examples with 8 everyday objects with various motions applied, and through technical evaluation to measure the accuracy of motion reconstruction.

2020

1. 

   Romeo: A Design Tool for Embedding Transformable Parts in 3D Models to Robotically Augment Default Functionalities

   Jiahao Li, Meilin Cui, Jeeeun Kim, and 1 more author

   In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technolog (UIST ’20), Sep 2020

   Abs PDF

   Reconfiguring shapes of objects enables transforming existing passive objects with robotic functionalities, e.g., a transformable coffee cup holder can be attached to a chair’s armrest, a piggy bank can reach out an arm to ’steal’ coins. Despite the advance in end-user 3D design and fabrication, it remains challenging for non-experts to create such ’transformables’ using existing tools due to the requirement of specific engineering knowledge such as mechanisms and robotic design. We present Romeo – a design tool for creating transformables to robotically augment objects’ default functionalities. Romeo allows users to transform an object into a robotic arm by expressing at a high level what type of task is expected. Users can select which part of the object to be transformed, specify motion points in space for the transformed part to follow and the corresponding action to be taken. Romeo then automatically generates a robotic arm embedded in the transformable part ready for fabrication. A design session validated this tool where participants used Romeo to accomplish controlled design tasks and to open-endedly create coin-stealing piggy banks by transforming 3D objects of their own choice.

2019

1. 

   Robiot: A Design Tool for Actuating Everyday Objects with Automatically Generated 3D Printable Mechanisms

   Jiahao Li, Jeeeun Kim, and Xiang ’Anthony’ Chen

   In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST ’19), Sep 2019

   Abs PDF

   Users can now easily communicate digital information with an Internet of Things; in contrast, there remains a lack of support to automate physical tasks that involve legacy static objects, e.g. adjusting a desk lamp’s angle for optimal brightness, turning on/off a manual faucet when washing dishes, sliding a window to maintain a preferred indoor temperature. Automating these simple physical tasks has the potential to improve people’s quality of life, which is particularly important for people with a disability or in situational impairment.We present Robiot – a design tool for generating mechanisms that can be attached to, motorized, and actuating legacy static objects to perform simple physical tasks. Users only need to take a short video manipulating an object to demonstrate an intended physical behavior. Robiot then extracts requisite parameters and automatically generates 3D models of the enabling actuation mechanisms by performing a scene and motion analysis of the 2D video in alignment with the object’s 3D model. In an hour-long design session, six participants used Robiot to actuate seven everyday objects, imbuing them with the robotic capability to automate various physical tasks.