I can't find that passage, but I remember someone (can't remember who) in one of the books (can't remember which one!) says that even though Muggles can't see them, they can still feel their presence. Also, Dudley did say that he remembered things when he encountered them, and judging by the state he was left in, he did feel that they were there.

Sarah x