A few thoughts. You've probably thought of them already but I'm going to rabbit on anyway
so get yourself one of these:
I've used a couple of different editing services for my first two novels (but only for the first 3 chapters to get them agent ready), and found the service varied greatly. But I picked
cheaper editing services, which in my view was a mistake. I went for price not editor. I think you'll get what you pay for. It hasn't put me off though, and if I were to self-publish I would def make use of someone, I'd just be pickier about who. I used a slightly different editing service for the first ten pages of novel three and it was immensely useful and actually helped me develop my new MC (Mia).
I think the quality of the feedback depends on the editor. And the price, like you already mention. Someone with a low fee might not have the incentive to put your MS through the mill, while someone with a higher fee should. A high fee, though, doesn't necessarily guarantee quality.
I'd check out their credentials too. As with everything, they matter.
As for why not a beta reader only? Publishing houses use editors for a reason. If you self-publish, you're your own publishing house. If I were you, I'd publish to the same spec as the p house. Why do any less?
A professional editor has an incentive to give you good feedback. It's their livelihood. You might use them for the next book. A beta reader may wish you well of course, but they don't have the same incentive (that's not a judgment, btw). They're just doing you a favour. A good editor will think deeply which takes time, but beta readers might not spend hours pondering just one chapter (again, that's not a criticism, purely the reality that when we do stuff for free we do it differently). I also think a good editor has a different eye as well as the experience to see the nitty gritty which a beta reader might not have. And a good editor knows the market, genre expectations, they have seen similar works etc.
There are some WONDERFUL beta readers (on this site especially). Total gems who read with high skill. But I can't expect them to read to the kinds of depths as I'd like them to, and then keep going back and forth with my fixes. Like you say, it's so time consuming and an imposition to expect that kind of work for free which is why an editor can be good. Yes, we trade, but I can't expect a friend to go to the same lengths, even if it is reciprocal.
As for this:
I have also seen a couple of samples here that have been professionally edited which beggar belief and suggest court action might be in order and I am no editor.
Three things:
1) an editor can
help someone's work, they can't
save someone's work.
2) just because someone had a crit doesn't mean they can, or want to, sort what the editor has flagged.
3) that novel might not sell because it doesn't fit with current markets
From how I understand it there're several types of editors: structural, developmental, copy etc, you probably know more about that. You probably wouldn't need all.
If you won millions in the lottery, would you get an editor?
Of course, an editor isn't always right.
The guy on pop-up panel yesterday, John, said he uses a several (I think he said) types of editors. You could get in touch with him and ask who. John has a fair amount of success with his work, so....
You're self-publishing. Personally, I'd give it the best treatment possible, because the work will be out there for eternity. And well, it will cost you, but to build a loyal readership I'd do what is necessary (which includes a good cover designer).