The spec for real ballast is about 2in or so single-size stone, with no “crisps” or “carrots” present. That’s why I have used ballast that is labelled for N gauge for years.
Modellers’ ballast is normally too bright when laid – except when one wants to represent new trackage. Once set it lends itself easily to being toned down with well-diluted paint colours. I do it in several stages: one can always add more paint though one cannot easily take it away.
Experimentally, I have just painted some track (rails and chairs “rust” colour) before laying it, with the idea of ballasting it with fine sawdust (i.e. powder) and spraying it a dark grey mist so as to represent that found in a colliery network.