The thing about tea is that the flavor characteristics we tend to like (essential oils, etc) come out first during an infusion. The things we tend not to like (tannins, etc) come out later. Tannins typically result in a "bitter" or "astringent" taste. It's a "chewy" sensation toward the back of your mouth. Whether we're talking about tea, coffee or wine, a little tannin can be very good. A lot almost never is.
If you are experiencing this in your ganache, the most likely reason is not any type of reaction with the cream. Rather, you are most likely over-infusing your tea into the cream. Generally speaking, there are three ways to pull back on this. First, infuse the tea in cream for less time. If there's any way to infuse the tea for a shorter period of time and still get the final flavor you want, this is the way to go. Second, lower temperatures. Heat brings out the tannins. Turn the heat down. Third, surface area. Some teas have large leaves, some have small. The smaller the leaves, the faster the tea will infuse and the faster those tannins will rush into your cream.
As a side note, Earl Grey gets its distinctive flavor from oil of bergamot. If you add this oil directly rather than infusing tea into the cream you might have more luck, along with more control over the flavor profile of the final ganache. Just a thought.