Mr Sarwar said "the leadership in Downing Street has to change".