Objective To investigate the status of self-management ability in elderly patients with type 2 diabetes mellitus (T2DM) in community, and to analyze the influencing factors using structural equation model. Methods A convenient sampling method was used to investigate patients aged 65 years and above with T2DM in Hongkou district of Shanghai. The patients were assessed using diabetes self-management questionnaire (DSMQ), Beck depression inventory (BDI), diabetes management self-efficacy scale (DMSES), and diabetes distress scale (DDS). Spearman correlation analysis was conducted to explore the relationship among the variables. A structural equation model was constructed to analyze the effect of depression, self-efficacy, and diabetes distress on self-management levels among the patients. Results A total of 588 elderly patients with T2DM were included in this survey, the DSMQ score was 32.00 (28.00, 38.00), the DMSES score was 158.50 (135.25, 172.00), the BDI score was 0.00 (0.00, 2.00), and the DDS score was 23.00 (18.00, 31.00). The final model demonstrated good fit to the sample data, self-efficacy had a positive impact on self-management levels (β＝0.558, P＜0.001), and depression had a positive impact on diabetes distress (β＝0.599, P＜0.001). Conversely, depression had a negative impact on self-efficacy (β＝－0.314, P＜0.001), and diabetes distress also had a negative impact on self-efficacy (β＝－0.267, P＜0.001). Self-efficacy had the greatest impact on self-management level (the total effect was 0.558), and it was a direct impact. Depression (the total effect was －0.264) and diabetes distress (the total effect was －0.149) had indirect effects. Self-efficacy played a mediating role between depression and self-management, with an indirect effect of －0.175, accounting for 66.3% of the total effect. Self-efficacy and diabetes distress played a chain mediating role between depression and self-management, with an indirect effect of －0.089, accounting for 33.7% of the total effect. Conclusion Self-efficacy has the greatest effect on the self-management level of elderly patients with diabetes mellitus. Depression and diabetes distress indirectly affect self-management levels through the mediating role of self-efficacy. Community and clinical healthcare workers should focus on enhancing the self-efficacy of patients with diabetes mellitus and implementing targeted interventions for their symptoms of depression and diabetes distress, so as to improve their self-management capabilities.