I don't think they said not to. I think they were saying it is a term often misunderstood and it really means writing style. So when an agent talks about looking for the voice that's what they mean: do you have a writing style that sets you apart.
One thing that struck a chord with me was when she said sometimes she reads the query letter, and the writing just hooks her, regardless of the topic. I read some books where I don't really care what the plot is, I'm just enjoying the process.