I understand the compulsion to assume the worst and focus on the negative because I do that myself, but I'm sure it's not the case that he can't think of anything other than you being "smart" or "pretty." Personally, I have a really hard time describing to people exactly why I like them and when people expect me to do so it makes me extremely uncomfortable and I feel it's a bit unfair; someone's inability to articulate what they like about you beyond what you may feel are "generic" reasons (smart, funny, kind, pretty, etc.) does not mean there is nothing else to like. I'd say this is more indicative of how you feel about yourself rather than how he feels about you — it's unwise to do his thinking for him by making such assumptions.
Keep this in mind as well: the fact that he recognized that his joke fell flat and made you feel uncomfortable and subsequently apologized is not insignificant.