Hola, and my own soapbox.
In this day and age, anything that gets folks out and doing something remotely social is worth any impact to our environment. If you really want to make a difference, get the giant scale offenders to change, rather than the few ignorant individuals. I know the early stages of GeoCaching for me started in scouts with basic orientation drills and a marker at the end of some complex runs.
I have done it a few times in a couple parks in the eastern Subs, and once up in Plymouth near my own home. It is great to get outside and just take in the environment. Now it is not for everybody, my first and only attempt to take a date on a find, did not go well. (Though she liked hiking, it was the off the path, thorns, and hills, that turned it into a bad choice).
But as far as hobbies go, if you are respectful of the environment it is something you will do again. If not, I am guessing you would not do it more than a couple times for a bottlecap, or arcade token..