The Black working class audience for jazz remained healthy at least until the late '60s, and in some ways into the '70s -- and it still exists here in Detroit, though not to the degree it once did. In any case, here's the great Gerald Early talking about Jimmy Smith et. al. in the '60s in an interview with Ethan Iverson,
"You know, when I was growing up, somebody like Jimmy Smith was like God. To black people he was God! It was like a Hammond B-3 was the thing! If Jimmy Smith was God, then he had some acolytes under him, like Richard “Groove” Holmes, and Brother Jack McDuff, and these people. And I didn’t know any white people who really liked this music very much. I’m sure there were some because Jimmy Smith was pretty popular, but by and large this was black music.
"At any rate, while they certainly had their detractors and there were some who didn’t like that sound, their importance culturally in the African-American community was quite apart from whatever their worth was as musicians. Those that had this groove style of playing (especially with a Hammond organ) occupied a kind of position that I think was quite apart from how any professional musicians might evaluate them based purely on their abilities as musicians. What they were expressing as musicians was deeply connected to the culture.
"And I think that it’s important for any musician that is interested in jazz – or anyone who really wants to understand this music – to understand that aspect of the musicians as well. How the first-generation fans decide how they’re going to translate the music into their cultural lives may have nothing to do with how later listeners see it or what they think it’s about."