One question I've been concerning myself with is why Kerouac's On the Road is so much better than everything else he wrote.
There is a clear biographical answer. The "scroll version" he wrote in two weeks on coffee (not Benzedrine, as the myth has it) was actually like the fifth time he'd written up the material, and he was transcribing very well worked out and commented on by friends routines at that point. Then he massively rewrote the scroll version once and then his rewrite was massively rewritten again by an editor in the tradition of Max Perkins (who effectively co-wrote Fitzgerald, Hemingway, and Wolfe).
Kerouac himself began to believe the myth of "spontaneous bop prosody," thinking the first write was in some way holy. While this worked really well for his unpublished until after he died Visions of Cody, which consists in a bunch of poetic sketches, his narrative stuff after Road is much worse for not being rewritten.
But this does not answer the question. What is it about, say, Dharma Bums that makes it so much worse than On the Road? I think I've figured this out.
On the Road is very impressionistic. The narrator does not reveal very much of his inner life. He just tells you everything that is going on with the people, events, and places around him. We really only get a view of him in terms of how people react to him (e.g. when the Joan Burroughs character, I forget her in text name, says "Same old Sal" after he does something particularly naive). But somehow by the end of the novel there is a sadness in what he writes. He realizes that Cassidy is not really a saint, but he still can't stop thinking about him and the promise he once held. This, plus the strange spiritual stuff (i.e. his dream of the snake that he tells people about, etc) is in part why it is the great American novel, we see the narrator go from innocence to experience.
With Dharma Bums you constantly get the narrator's editorializing about what is going on. The feeling is that the writer is narrating here, not the narrator who is acting in the fictitious universe. And some of it is deluded. Unlike with Road, there is no distance between the writer and the narrator. Maybe if he had used everybody's real names (i.e. Gary Snyder instead of the ludicrous "Japhy Ryder") this would not have made everything ring as falsely as it does from a novelistic perspective. It's maddening, because parts of Dharma Bums is brilliant. What you have is a fantastic, albeit extremely rough draft about somebody desperately trying to embrace Buddhism to try to deal with the tragic aspects of life, but it doesn't really work and he still has to drink himself insensible. With On the Road, the writer was aware that the promised liberation of Cassidy et. al. wasn't going to make up for the tragic aspects of life, but through rewriting he very skillfully shows the narrator slowly coming to grasp this.
Kerouac was always too smart and too attuned to the tragic to buy into the liberatory promises of either the beatniks or the hippies (and Dharma Bums was part of the hippie canon). But he bought into his own bullshit about the writing process, one that totally misdescribed how his masterpiece was written. Given the man's talents, and how this led his other books (e.g. Desolation Angels) to completely degenerate into hastily scrawled, psychologically obtuse diary entries, this in itself is a pretty big tragedy I think.