I don't think what you are feeling is unusual. Many of us do on different levels. For me, it is usually when I feel the cache may have been placed without those nearby knowing about it. It is always a good thing for these urban caches to state in the description that permission has been granted.

General passersby do not bother me as much. I make a quick assumption about anyone watching me and back off or go ahead and get the cache anyway depending on what I think.

I much prefer the thought that a fox or bear or deer is lurking in the woods watching me go by. Animals are much more predictable than humans!