When I was little we lived in France. I spoke French at school and English at home. I could keep up just fine. I understood everyone and they understood me. However, idioms were not my thing. My fourth grade teacher told my mom that's how she could tell I wasn't French; I didn't have an accent but sometimes you could tell I was translating things directly over from English in my head. It was the way of saying things, not the saying them that was hard.
Culture gives us the pictures for things. When you want to explain something, you put the picture into words and hand it off. You learn the way of saying the picture and that's an idiom.
I'm not sure how much of the confusion was due to idioms and how much was just because I'm not exactly one to think along the same lines as most people.
At any rate, thank goodness I had my parents to teach me English idioms, or I wouldn't have had anything to translate.