Yes, it's probably safe. And is, probably aggregate picked up in and around the quarry boundaries.
But, a little prudence can certainly go a long way in sorting these things out, hmmmm?
So, what would I do? WWAD?.....
- Place one (?) of the affected rocks in a bucket or tub. Let it soak for a couple days. Then vigorously scrub and scrape at the aggregate area. is it rock? Or, REALLY hardened clay/caliche deposits on your Texas Holey Rock? If that stuff just doesn't come off of the rock after that? Good! It belongs there.