People have long imagined intelligent home service robots effortlessly handling daily chores like folding T-shirts or neatly hanging skirts. Yet creating a general-purpose robot capable of manipulating diverse clothes remains a significant challenge. Clothes, inherently deformable, pose difficulties due to their complex and changing shapes. Although recent advances have improved robots’ abilities to fold and flatten clothes, these skills often only apply to specific clothes or tasks. Our research addresses this issue through a novel representation of clothes—semantic keypoints, which capture the essential features of clothes through some sparse keypoints with semantic meaning. Integrating semantic keypoints with foundation models, we develop a general-purpose clothes manipulation method, which can be applied to a wide variety of clothes categories and manipulation tasks.

CLASP:

Y. Deng, and D. Hsu. General-purpose clothes manipulation with semantic keypoints. In IEEE Int. Conf. on Robotics & Automation, 2025. 
PDF

Clothes manipulation is a critical capability for home service robots; yet,  existing methods are often confined to specific tasks, such as folding or flattening, due to the complex high-dimensional geometry of deformable fabric. This paper presents CLothes mAnipulation with Semantic keyPoints (CLASP) for general-purpose clothes manipulation, which enables the robot to perform diverse manipulation tasks over different types of clothes.  The key idea of CLASP is semantic keypoints—e.g., ”right shoulder”, ”left sleeve”, etc.—a sparse spatial-semantic representation that is salient for both perception and action. Semantic keypoints of clothes can be effectively extracted from depth images and are sufficient to represent a broad range of clothes manipulation policies. CLASP leverages semantic keypoints to bridge LLM-powered task planning and low-level action execution in a two-level hierarchy. Extensive simulation experiments show that CLASP outperforms baseline methods across diverse clothes types in both seen and unseen tasks. Further, experiments with a dual-arm system on four distinct tasks—folding, flattening, hanging, and placing—confirm CLASP’s performance on a real robot.