I'm facing a similar dilemma, for I wrote my first novel in 2014, then spent the first seven months of 2015 querying literary agents. I approached 160 of them, and it took a while for me to realise that I'd made a beginner's mistake in writing a story that was 176,000 words long—almost double the acceptable length for a debut novel by an unknown author.
There was no way that I could hack it down to the correct length, though removing repeated and filler words reduced it to 160,000. I still had faith in the story, and my beta-readers loved it, so decided to hang onto it for use as a third or fourth novel in a series about my Cornish detective—it's better to travel hopefully, even if you don't know where you're going!
I decided to write a prequel to the first novel, of the correct length and with a simpler plot containing the attention-grabbing elements of naturism, BDSM and smuggling. This caused me unanticipated technical problems, for I had to reintroduce the key players again, meaning I needed to tinker with the first manuscript. I'm on the last chapter of the second novel and have lots of ideas coming to me for a third story in the series, with tentative plans for a fourth.
I've become bound up in the life of my protagonist detective and his investigations, though I originally wrote a psychological thriller as I saw it was a genre that sold well. I'd been intending to write a literary novel about relationships, dating and how technology affects the way we communicate but realised that literary novels are hard to sell even for established authors. I certainly didn't think genre writing was dumbing down and made my characterisation and plotting complex.
I'll probably query the literary agents who had the manners to give me a personalised rejection, with the second novel. I've become more commercially aware in the last two years, knowing what's marketable to modern tastes. There's still self-publishing ebooks, but I already have 44 on Smashwords and Amazon, which have disappeared in the ocean of fresh titles appearing every day. If I had a twin brother I'd get him marketing them and me non-stop, but I don't have the time.
Anything to do with book publishing is unpredictable—no one knows what will work until it does! We shouldn't put obstacles in our own way, as there are hurdles enough to leap. I think that my chances of finding a readership are higher with a series of stories featuring the same detective, set in a beautiful location where he solves intriguing crimes. I've read of authors finding representation for their first novel, only to be panicked by their agent asking them if there are more books already written, especially if it has the makings of a series.
I may be waiting for a train that never arrives, but at least I'll have matching luggage to take on the journey if it steams into the station.