Should I really Worry about my weight?

If I'm twenty pounds over weight, should I really worry about it? What health factors should I consider and should I try to loose twenty pounds? I don't have any health issues and I'm comfortable at this weight.