James reacts to housing minister suggesting the 'unelected Labour Party' voted against Tory plans