2024

1. 

   Towards Assessing Compliant Robotic Grasping From First-Object Perspective via Instrumented Objects

   Maceon Knopke, Liguo Zhu, Peter Corke, and Fangyi Zhang

   IEEE Robotics and Automation Letters (RA-L), 2024

   Abs HTML PDF Video

   Grasping compliant objects is difficult for robots - applying too little force may cause the grasp to fail, while too much force may lead to object damage. A robot needs to apply the right amount of force to quickly and confidently grasp the objects so that it can perform the required task. Although some methods have been proposed to tackle this issue, performance assessment is still a problem for directly measuring object property changes and possible damage. To fill the gap, a new concept is introduced in this paper to assess compliant robotic grasping using instrumented objects. A proof-of-concept design is proposed to measure the force applied on a cuboid object from a first-object perspective. The design can detect multiple contact locations and applied forces on its surface by using multiple embedded 3D Hall sensors to detect deformation relative to embedded magnets. The contact estimation is achieved by interpreting the Hall-effect signals using neural networks. In comprehensive experiments, the design achieved good performance in estimating contacts from each single face of the cuboid and decent performance in detecting contacts from multiple faces when being used to evaluate grasping from a parallel jaw gripper, demonstrating the effectiveness of the design and the feasibility of the concept.

2. 

   Learning fabric manipulation in the real world with human videos

   Robert Lee, Jad Abou-Chakra, Fangyi Zhang, and Peter Corke

   In IEEE International Conference on Robotics and Automation (ICRA), 2024

   Abs HTML PDF

   Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Learning approaches stand out as promising for this domain as they allow us to learn behaviours directly from data. Most prior methods however rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects or rely on large datasets. A promising alternative is to learn fabric manipulation directly from watching humans perform the task. In this work, we explore how demonstrations for fabric manipulation tasks can be collected directly by humans, providing an extremely natural and fast data collection pipeline. Then, using only a handful of such demonstrations, we show how a pick-and-place policy can be learned and deployed on a real robot, without any robot data collection at all. We demonstrate our approach on a fabric smoothing and folding task, showing that our policy can reliably reach folded states from crumpled initial configurations. Code, video and data are available on the project website: https://sites.google.com/view/foldingbyhand

3. 

   All-Optical Diffractive Deep Neural Networks Enabled Laser-Reduced Graphene Oxide Tactile Sensor for Braille Recognition

   Xing Liu, Li Fang, Fangyi Zhang, Qiwen Zhang, Zhengfen Wan, and 1 more author

   ACS Applied Electronic Materials, Mar 2024

2023

1. 

   Doping of Laser-Induced Graphene and Its Applications

   Qiwen Zhang, Fangyi Zhang, Xing Liu, Zengji Yue, Xi Chen, and 1 more author

   Advanced Materials Technologies, Mar 2023

   Abs

   Abstract Laser-induced graphene (LIG) has attracted extensive attention owing to its facile preparation of graphene and direct engraving patterns for devices. Various applications are demonstrated such as sensors, supercapacitors, electrocatalysis, batteries, antimicrobial, oil and water separation, solar cells, and heaters. In recent years, doping has been employed as a significant strategy to modulate the properties of LIG and thereby improve the performance of LIG devices. Due to the patternable manufacture, controllable morphologies, and the synergistic effect of doped atoms and graphene, the doped LIG devices exhibit a high sensitivity of sensing, pseudocapacitance performance, and biological antibacterial. This paper reviews the latest novel research progress of heteroatom and nanoparticles doped LIG in synthesis, properties, and applications. The fabrications of LIG and typical doping approaches are presented. Special attention is paid to two doping processes of LIG: the one-step laser irradiation method and the two-step laser modification consisting of deposition, drop-casting, and duplicated laser pyrolysis. Doped LIG applications with improved performance are mainly highlighted. Taking advantage of doped LIG’s properties and device performances will provide excellent opportunities for developing artificial intelligence, data storage, energy, health, and environmental applications.

2. 

   Laser-scribed graphene for sensors: preparation, modification, applications, and future prospects

   Xing Liu, Fangyi Zhang, Qiwen Zhang, Zhengfen Wan, and Xi Chen

   Light: Advanced Manufacturing, Apr 2023

3. 

   Re-Evaluating Parallel Finger-Tip Tactile Sensing for Inferring Object Adjectives: An Empirical Study

   Fangyi Zhang, and Peter Corke

   In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Apr 2023

   Abs PDF

   Finger-tip tactile sensors are increasingly used for robotic sensing to establish stable grasps and to infer object properties. Promising performance has been shown in a number of works for inferring adjectives that describe the object, but there remains a question about how each taxel contributes to the performance. This paper explores this question with empirical experiments, leading insights for future finger-tip tactile sensor usage and design.

4. 

   A Linkage-based Doubly Imbalanced Graph Learning Framework for Face Clustering

   Huafeng Yang, Qijie Shen, Xingjian Chen, Fangyi Zhang, and Rong Du

   In SIAM International Conference on Data Mining, Apr 2023

   Abs PDF

   In recent years, benefiting from the expressive power of Graph Convolutional Networks (GCNs), significant breakthroughs have been made in face clustering area. However, rare attention has been paid to GCN-based clustering on imbalanced data. Although imbalance problem has been extensively studied, the impact of imbalanced data on GCN- based linkage prediction task is quite different, which would cause problems in two aspects: imbalanced linkage labels and biased graph representations. The former is similar to that in classic image classification task, but the latter is a particular problem in GCN-based clustering via linkage prediction. Significantly biased graph representations in training can cause catastrophic over-fitting of a GCN model. To tackle these challenges, we propose a linkage-based doubly imbalanced graph learning framework for face clustering. In this framework, we evaluate the feasibility of those existing methods for imbalanced image classification problem on GCNs, and present a new method to alleviate the imbalanced labels and also augment graph representations using a Reverse-Imbalance Weighted Sampling (RIWS) strategy. With the RIWS strategy, probability-based class balancing weights could ensure the overall distribution of positive and negative samples; in addition, weighted random sampling provides diverse subgraph structures, which effectively alleviates the over-fitting problem and improves the representation ability of GCNs. Extensive experiments on series of imbalanced benchmark datasets synthesized from MS-Celeb-1M and DeepFashion demonstrate the effectiveness and generality of our proposed method.

2022

1. 

   Robust Graph Structure Learning via Multiple Statistical Tests

   Yaohua Wang, Fangyi Zhang, Ming Lin, Senzhang Wang, Xiuyu Sun, and 1 more author

   In Advances in Neural Information Processing Systems (NeurIPS), Apr 2022

   Abs PDF Code

   Graph structure learning aims to learn connectivity in a graph from data. It is particularly important for many computer vision related tasks since no explicit graph structure is available for images for most cases. A natural way to construct a graph among images is to treat each image as a node and assign pairwise image similarities as weights to corresponding edges. It is well known that pairwise similarities between images are sensitive to the noise in feature representations, leading to unreliable graph structures. We address this problem from the viewpoint of statistical tests. By viewing the feature vector of each node as an independent sample, the decision of whether creating an edge between two nodes based on their similarity in feature representation can be thought as a single statistical test. To improve the robustness in the decision of creating an edge, multiple samples are drawn and integrated by multiple statistical tests to generate a more reliable similarity measure, consequentially more reliable graph structure. The corresponding elegant matrix form named -Attention is designed for efficiency. The effectiveness of multiple tests for graph structure learning is verified both theoretically and empirically on multiple clustering and ReID benchmark datasets.

2. 

   Ada-NETS: Face Clustering via Adaptive Neighbour Discovery in the Structure Space

   Yaohua Wang, Yaobin Zhang, Fangyi Zhang, Senzhang Wang, Ming Lin, and 2 more authors

   In International Conference on Learning Representations (ICLR), Apr 2022

   Abs PDF Code

   Face clustering has attracted rising research interest recently to take advantage of massive amounts of face images on the web. State-of-the-art performance has been achieved by Graph Convolutional Networks (GCN) due to their powerful representation capacity. However, existing GCN-based methods build face graphs mainly according to kNN relations in the feature space, which may lead to a lot of noise edges connecting two faces of different classes. The face features will be polluted when messages pass along these noise edges, thus degrading the performance of GCNs. In this paper, a novel algorithm named Ada-NETS is proposed to cluster faces by constructing clean graphs for GCNs. In Ada-NETS, each face is transformed to a new structure space, obtaining robust features by considering face features of the neighbour images. Then, an adaptive neighbour discovery strategy is proposed to determine a proper number of edges connecting to each face image. It significantly reduces the noise edges while maintaining the good ones to build a graph with clean yet rich edges for GCNs to cluster faces. Experiments on multiple public clustering datasets show that Ada-NETS significantly outperforms current state-of-the-art methods, proving its superiority and generalization.

3. 

   Jmpnet: Joint Motion Prediction for Learning-Based Video Compression

   Dongyang Li, Zhenhong Sun, Zhiyu Tan, Xiuyu Sun, Fangyi Zhang, and 2 more authors

   In IEEE International Conference on Acoustics, Speech and Signal Processing, Apr 2022

   Abs PDF

   In recent years, more attention is attracted by learning-based approaches in the field of video compression. Recent methods of this kind normally consist of three major components: intra-frame network, motion prediction network, and residual network, among which the motion prediction part is particularly critical for video compression. Benefiting from the optical flow which enables dense motion prediction, recent methods have shown competitive performance compared with traditional codecs. However, problems such as tail shadow and background distortion in the predicted frame remain unsolved. To tackle these problems, JMPNet is introduced in this paper to provide more accurate motion information by using both optical flow and dynamic local filter as well as an attention map to further fuse these motion information in a smarter way. Experimental results show that the proposed method surpasses state-of-the-art (SOTA) rate-distortion (RD) performance in the most data-sets.

2021

1. 

   GCN-Based Linkage Prediction for Face Clustering on Imbalanced Datasets: An Empirical Study

   Huafeng Yang, Xingjian Chen, Fangyi Zhang, Guangyue Hei, Yunjie Wang, and 1 more author

   In Workshops of the International Joint Conference on Artificial Intelligence, Apr 2021

   Abs PDF Code

   In recent years, benefiting from the expressivepower of Graph Convolutional Networks (GCNs),significant breakthroughs have been made in faceclustering. However, rare attention has been paidto GCN-based clustering on imbalanced data. Al-though imbalance problem has been extensivelystudied, the impact of imbalanced data on GCN-based linkage prediction task is quite different,which would cause problems in two aspects: im-balanced linkage labels and biased graph represen-tations. The problem of imbalanced linkage labelsis similar to that in image classification task, but thelatter is a particular problem in GCN-based clus-tering via linkage prediction. Significantly biasedgraph representations in training can cause catas-trophic overfitting of a GCN model. To tacklethese problems, we evaluate the feasibility of thoseexisting methods for imbalanced image classifica-tion problem on graphs with extensive experiments,and present a new method to alleviate the imbal-anced labels and also augment graph representa-tions using a Reverse-Imbalance Weighted Sam-pling (RIWS) strategy, followed with insightfulanalyses and discussions. A series of imbalancedbenchmark datasets synthesized from MS-Celeb-1M and DeepFashion will be openly available.

2. 

   Interpolation Variable Rate Image Compression

   Zhenhong Sun, Zhiyu Tan, Xiuyu Sun, Fangyi Zhang, Yichen Qian, and 2 more authors

   In ACM International Conference on Multimedia, Virtual Event, China, Apr 2021

   Abs PDF

   Compression standards have been used to reduce the cost of image storage and transmission for decades. In recent years, learned image compression methods have been proposed and achieved compelling performance to the traditional standards. However, in these methods, a set of different networks are used for various compression rates, resulting in a high cost in model storage and training. Although some variable-rate approaches have been proposed to reduce the cost by using a single network, most of them brought some performance degradation when applying fine rate control. To enable variable-rate control without sacrificing the performance, we propose an efficient Interpolation Variable-Rate (IVR) network, by introducing a handy Interpolation Channel Attention (InterpCA) module in the compression network. With the use of two hyperparameters for rate control and linear interpolation, the InterpCA achieves a fine PSNR interval of 0.001 dB and a fine rate interval of 0.0001 Bits-Per-Pixel (BPP) with 9000 rates in the IVR network. Experimental results demonstrate that the IVR network is the first variable-rate learned method that outperforms VTM 9.0 (intra) in PSNR and Multiscale Structural Similarity (MS-SSIM).

2019

1. 

   Adversarial discriminative sim-to-real transfer of visuo-motor policies

   Fangyi Zhang, Jürgen Leitner, Zongyuan Ge, Michael Milford, and Peter Corke

   The International Journal of Robotics Research (IJRR), Apr 2019

   Abs PDF Code

   Various approaches have been proposed to learn visuo-motor policies for real-world robotic applications. One solution is first learning in simulation then transferring to the real world. In the transfer, most existing approaches need real-world images with labels. However, the labelling process is often expensive or even impractical in many robotic applications. In this paper, we propose an adversarial discriminative sim-to-real transfer approach to reduce the cost of labelling real data. The effectiveness of the approach is demonstrated with modular networks in a table-top object reaching task where a 7 DoF arm is controlled in velocity mode to reach a blue cuboid in clutter through visual observations. The adversarial transfer approach reduced the labelled real data requirement by 50%. Policies can be transferred to real environments with only 93 labelled and 186 unlabelled real images. The transferred visuo-motor policies are robust to novel (not seen in training) objects in clutter and even a moving target, achieving a 97.8% success rate and 1.8 cm control accuracy.

2017

1. 

   The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research

   Jürgen Leitner, Adam W. Tow, Niko Sünderhauf, Jake E. Dean, Joseph W. Durham, and 13 more authors

   In IEEE International Conference on Robotics and Automation (ICRA), Apr 2017

   Abs PDF

   Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark (APB). Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of \emphcomplete robotic systems – including perception and manipulation – instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.

2. 

   Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies

   Fangyi Zhang, Jürgen Leitner, Michael Milford, and Peter Corke

   In Australasian Conference on Robotics and Automation (ACRA), Dec 2017

   Abs PDF

   While deep learning has had significant successes in computer vision thanks to the abundance of visual data, collecting sufficiently large real-world datasets for robot learning can be costly. To increase the practicality of these techniques on real robots, we propose a modular deep reinforcement learning method capable of transferring models trained in simulation to a real-world robotic task. We introduce a bottleneck between perception and control, enabling the networks to be trained independently, but then merged and fine-tuned in an end-to-end manner to further improve hand-eye coordination. On a canonical, planar visually-guided robot reaching task a fine-tuned accuracy of 1.6 pixels is achieved, a significant improvement over naive transfer (17.5 pixels), showing the potential for more complicated and broader applications. Our method provides a technique for more efficient learning and transfer of visuo-motor policies for real robotic systems without relying entirely on large real-world robot datasets.

3. 

   Tuning Modular Networks With Weighted Losses for Hand-Eye Coordination

   Fangyi Zhang, Jürgen Leitner, Michael Milford, and Peter Corke

   In IEEE Conference on Computer Vision and Pattern Recognition Workshops, Jul 2017

   Abs PDF

   This paper introduces an end-to-end fine-tuning method to improve hand-eye coordination in modular deep visuo-motor policies (modular networks) where each module is trained independently. Benefiting from weighted losses, the fine-tuning method significantly improves the performance of the policies for a robotic planar reaching task.

2016

1. 

   Let the light guide us: VLC-based localization

   Kejie Qiu, Fangyi Zhang, and Ming Liu

   IEEE Robotics & Automation Magazine, Jul 2016

   Abs PDF

   We propose to use visible-light beacons for low-cost indoor localization. Modulated light-emitting diode (LED) lights are adapted for localization as well as illumination. The proposed solution consists of two components: light-signal decomposition and Bayesian localization.

2015

1. 

   Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control

   Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Upcroft, and Peter Corke

   In Australasian Conference on Robotics and Automation (ACRA), Dec 2015

   Abs PDF

   This paper introduces a machine learning based system for controlling a robotic manipulator with visual perception only. The capability to autonomously learn robot controllers solely from raw-pixel images and without any prior knowledge of configuration is shown for the first time. We build upon the success of recent deep reinforcement learning and develop a system for learning target reaching with a three-joint robot manipulator using external visual observation. A Deep Q Network (DQN) was demonstrated to perform target reaching after training in simulation. Transferring the network to real hardware and real observation in a naive approach failed, but experiments show that the network works when replacing camera images with synthetic images.

2. 

   Asynchronous blind signal decomposition using tiny-length code for visible light communication-based indoor localization

   Fangyi Zhang, Kejie Qiu, and Ming Liu

   In IEEE International Conference on Robotics and Automation (ICRA), Dec 2015

   Abs PDF

   Indoor localization is a fundamental capability for service robots and indoor applications on mobile devices. To realize that, the cost and performance are of great concern. In this paper, we introduce a lightweight signal encoding and decomposition method for a low-cost and low-power Visible Light Communication (VLC)-based indoor localization system. Firstly, a Gold-sequence-based tiny-length code selection method is introduced for light encoding. Then a correlation-based asynchronous blind light-signal decomposition method is developed for the decomposition of the lights mixed with modulated light sources. It is able to decompose the mixed light-signal package in real-time. The average decomposition time-cost for each frame is 20 ms. By using the decomposition results, the localization system achieves accuracy at 0.56 m. These features outperform other existing low-cost indoor localization approaches, such as WiFiSLAM.

3. 

   Visible light communication-based indoor localization using Gaussian process

   Kejie Qiu, Fangyi Zhang, and Ming Liu

   In IEEE/RSJ International Conference on Intelligent Robots and Systems, Dec 2015

   Abs PDF

   For mobile robots and position-based services, such as healthcare service, precise localization is the most fundamental capability while low-cost localization solutions are with increasing need and potentially have a wide market. A low-cost localization solution based on a novel Visible Light Communication (VLC) system for indoor environments is proposed in this paper. A number of modulated LED lights are used as beacons to aid indoor localization additional to illumination. A Gaussian Process(GP) is used to model the intensity distributions of the light sources. A Bayesian localization framework is constructed using the results of the GP, leading to precise localization. Path-planning is hereby feasible by only using the GP variance field, rather than using a metric map. Dijkstra’s algorithm-based path-planner is adopted to cope with the practical situations. We demonstrate our localization system by real-time experiments performed on a tablet PC in an indoor environment.

4. 

   Visible light communication-based indoor environment modeling and metric-free path planning

   Kejie Qiu, Fangyi Zhang, and Ming Liu

   In IEEE International Conference on Automation Science and Engineering, Dec 2015

   Abs PDF

   For mobile robots and position-based services, localization is the most fundamental capability while path-planning is an important application based on that. A novel localization and path-planning solution based on a low-cost Visible Light Communication (VLC) system for indoor environments is proposed in this paper. A number of modulated LED lights are used as beacons to aid indoor localization additional to illumination. A Gaussian Process (GP) is used to model the intensity distributions of the light sources. Path-planning is hereby feasible by using the GP variance field, rather than using a metric map. Graph-based path-planners are introduced to cope with the practical situations. We demonstrate our path-planning system by real-time experiments performed on a tablet PC in an indoor environment.