I get this all the time.
Something that has helped a lot is to have some visual aids for giving a sense of what I'm up to. I always have a few copies of invitations to previous exhibits in my camera bag, so people can see that I'm not A) a terrorist B) a peeping tom C) a commercial photographer who will make millions with images of someone's property and not share or D) a surveyor preparing plans to tear down the neighborhood and put in a superhighway.
In some cases, when i AM tresspassing or breaking laws (stopped by the sides of roads where you're not alowed to stop, wandering into construction sites, walking along catwalks of suspension bridges) I've found a nearly failsafe solution: an orange reflective vest ... the kind that surveyors and engineers wear. Cheap, light, and folds into a tiny package when you don't need it. People see a guy with a big tripod and an orange vest walking along bridge girders and think nothing of it. Cops included. (I have NOT tried this in post 9-11 new york, where I live now, and wouldn't recommend it).
Keep in mind that most people in the world have no idea what a large format camera is, or why someone would use one. So it's not really a good assumption that they'll know what you're doing. Especially with the kind of work that I do, which often involves subject matter that people don't think of as photogenic. They have no idea what I'm up to. And even if they do recognize you as a photographer, that won't be enough to put everyone at ease. Why are you photographing? The process can seem threatening or invasive to people when they don't understand it. If they know you're an artist and are interested in the shadow of the telephone pole, and not in their wives upstairs in the boudoir, they'll be more inclined to make your life easier.