People tend to overlook their parent's feelings when pushing them to downsize. The focus is usually on money and practicality. Valid thoughts, but not the whole picture. Maybe your angst is based on people telling you what you 'should' do.
I'm happy you're doing what you 'want' to do.