Activity recognition has been introduced as a means of detecting an action from a series of observations. Although in the literature, the terms “activity recognition” and “human activity recognition” are mostly used interchangeably, there exist several interesting applications for non-human subjects. In this work, we study animal activity recognition with special focus on horse gaits recognition. The on-body and unobtrusive system developed in this paper has several potential applications which can raise attention towards financial, emotional and veterinary health issues related to animals. Leveraging the pervasive use of smartwatches for activity tracking, we develop a smartwatch application to collect accelerometer data. The application is based on novel outlier detection and feature extraction techniques, in conjunction with state-of-the-art machine learning approaches. We perform real life experiments with two horses to evaluate the performances of our proposed system. To this aim, we place the monitoring device both on the horse saddle and the rider’s wrist. The results show a high accuracy in both scenarios, which allows a seamless and unobtrusive use of our wearable device application by the rider. In addition, we study the effects of sliding window size and sampling frequency, providing useful insights for future research in horse gaits recognition.