"All of us should take care, whether knowingly or unknowingly - and I don't believe he said it knowingly, nor is he likely to say it intentionally - to refrain from making such remarks," KC Venugopal said