The wide usage of wearable accelerometer-based activity trackers in recent years has provided a unique opportunity for in-depth research on physical activity (PA) and its relationship with health outcomes and interventions. Past analysis of activity tracker data relies heavily on aggregating minute-level PA records into day-level summary statistics, in which important information of diurnal PA patterns is lost. We propose a novel functional data analysis approach based on theory of Riemann manifolds for modeling PA records and longitudinal changes in PA temporal patterns. We model smoothed minute-level PA of a day as one-dimensional Riemann manifolds and longitudinal changes in PA in different visits as deformations between manifolds. With the proposed approach, we conduct comprehensive analyses on data from two clinical trials, RfH and MENU at UCSD, focusing on the effect of interventions on longitudinal changes in PA patterns and how different patterns of changes in PA influence weight loss, respectively. For both studies, important modes of variation in PA were identified to be significantly associated with lifestyle interventions/health outcomes.