The meme of asking an LLM how many R's are in the word strawberry went viral because of public ignorance about how LLMs work. An LLM doesn't even see the individual letters in a word. It takes its input in the form of word-sized chunks. As such, questions relating to individual letters, like asking it to count the number of letters or asking which words start with a specific letter, are uniquely difficult and completely unrelated to performance on other tasks in general.
If you think about it, the only way it can even infer strawberry has any r's at all is purely based on context clues from somewhere deep in its training set where someone may have spelled out the word "b-e-r-r-y" with dashes in some random blog or forum post. Therefore, the fact they even sometimes get these types of questions right should be considered nothing short of amazing.
