Prime Minister Mark Carney lost a cabinet minister over his oil pipeline agreement with Alberta.