On my caches, I've figured that most people are doing them when there is no ice on the lakes. Having said that, I've done many that I walked to. If you assume that the majority of cachers are getting the cache in the spring/summer/fall seasons, I would think that a 5 rating would be accurate.
Many caches can vary in difficulty. If you place a cache under heavy tree cover and someone finds it in early spring when there are no leaves, the difficulty goes down. I would say rate for the conditions that most people would experience when finding the cache.