Well, yes, but having been there in their youth doesn't mean that they would necessarily be passing by it during their adult lives. Quite aside from the fidelius charm, Grimmauld Place was a superb location for the order because it the most unlikely hideout for the Order. It was a house steeped in both dark magic and pureblood mania, which was against everything the Order stood for. Aside from which, the only person who had any link with the place was Sirius. Presuming that they did visit as a child, it would be plainly obvious that Sirius would not want to go anywhere near the place again. Why would they have even thought to look?
It's almost impossible to believe that Narcissa and Bellatrix hadn't been to 12 Grimmauld Place in their youth, being as it was the ancestral home of the family. Them not being able to see it as good as advertises the location to them. So even while the Order was there, conducting Order business, Bellatrix and Narcissa should have been able to *walk into the house* and even walk amongst the inhabitants, without seeing or hearing them, so long as the other occupants were there on Order business.
Personally, I am inclined to believe that Dumbledore did not visit the cottage after the Fidelius charm was cast. Since we know Sirius and Wormtail did, however, it's not beyond the bounds of reason that it was Sirius who passed the cloak from James to Dumbledore. Perhaps even Wormtail. That way James stays cooped up, Dumbledore gets his cloak, and nobody except those in on the secret would know ever see James, Lily or Harry. The only thing there which even remotely defies belief, is that James trusted his friends more than his headmaster. Of course given what we know from Lupin and Sirius about how much James trusted his friends, and given that they had found out relatively recently about Dumbledore and Grindlewauld, that is plausible for the character - if not particularly sensible in hindsight. Then again, had James been sensible, we'd probably just have finished reading
Bear in mind, too, that it's entire possible that Dumbledore may have been to the cottage in Godric's Hollow but unless he were "in on the secret" he wouldn't have been able to see James and Lily once they were in hiding. Flitwick says, for example, that the Dark Lord wouldn't have been able to see them "even if he had had his nose pressed upon their sitting room window".