I've not had any trouble with spices like ginger, cinnamon, cardamon, nutmeg etc seeping into the granite and transferring flavour. I've never tried chillies though.

I'd make your chocolate base, then transfer it to a mixer (even a kitchenaid), and do your oil based flavor additions there. Stainless steel is much easier to clean.

Thanks for the responses and thoughts. 

Chae, I have heard about infusing cocoa butter and adding. As this is specifically for drinking chocolate I'd rather not add any more cb. I'm making (at least) 72% chocolate so I'm happy with the amount of cb in there.

Lyndon, there are so many possibilities for additions!... I'm also tempted to just throw some unique and delicious things in the grinder.

Alas, I fear Sebastian is probably right that the stones will absorb some of these flavors/oils. Especially, the stronger ones (i.e., chiles and florals). I wonder if I can't get the result I want by first grinding (in a dry grinder) any additions and then adding to my chocolate at the end of its grind and let the two combine for a while in a separate mixer just with paddles... thereby furthering the conching process, as well.

Flavor contamination of your equipment is definitely going to be a challenge.  I'd consider dedicating equipment for that to avoid flavor transfer to your chocolates that you don't want to be flavored.

I grind all sorts of spices into my chocolate, tea leaves/ coffee beans too! I usually add them first with just a little cocoa beans or other oil to efficiently grind them, then some time later I'll add the rest of the cocoa beans or other ingredients

Most herbs and spices contain essential oils, which are soluble in fat. An option you might want to look into is infusing cocoa butter with the ingredients, then adding them into the melanger as its grinding. I remember in Elements of Desserts, Miyago made saffron-infused couverture this way.

I apologize if this has been specifically addressed... I looked for posts.

I'm wondering what issues I might deal with if I add some dried herbs or spices or chiles to the melanger while grinding my liquor? My final product will be drinking chocolate so texture is not really an issue since I will not be tempering. My goal is to create, on a larger scale, infused chocolate without having to infuse the milk every time I want to make a flavored drinking chocolate. (e.g., a lavender infused drinking chocolate without the step of milk infusion or adding crushed lavender). 

Any reason to think such spices would become infused in the stones in my grinder? (That wouldn't be so nice.)

Many thanks,

David