'It takes away a little money from people like me, who have done fantastically well over the decades', says Trevor Phillips.