I like the geography to be right and historical references to make sense (to me and to the characters) but for me those details need to be background to the story, not the story itself.

Yes, exactly. I do read non-fiction works on history and even geography sometimes, but to me a story is about characters, not about seeing how many obscure references you can fit in a fic