You are thinking about this too literally. What people can see is a representation of your life energy, and only the most skilled can make out more specific details other than your relative strength and whether you're a meister or a weapon. Your personal feelings on the matter notwithstanding, the ability to perceive souls is highly valued here. You can always choose not to exercise it, or ask permission first if you must use it.