This work introduces Siamese network into unmanned aerial manipulator (UAM) tracking, which is a model-free method and can be applied to track arbitrary objects. Considering severe object scale variation faced in practical UAM tracking application scenarios, a scale-aware Siamese network, i.e., SiamSA, is designed. Specifically, SiamSA employs a novel scale-channel attention strategy to excavate meaningful scale information of the object. Based on this strategy, scale attention network and scale-aware anchor proposal network are constructed to further improve the tracker’s robustness against scale variation. Besides, a new UAM tracking benchmark, namely UAMT100, is developed with 11 attributes and ~35,000 frames on a flying UAM platform. Exhaustive experiments on two authoritative aerial tracking benchmarks and UAMT100 benchmark validate the practicality and effectiveness of SiamSA with a real-time speed. Both the code and UAMT100 benchmark are now available.