It seems quite possible to us that in trying to ward off one problem you are creating another. You have to boil pastry cream, because it kills an enzyme in the eggs that otherwise causes the pastry cream to dissolve. But it can be hard to muster the fortitude to let it boil, because you're quite possibly thinking, "all these eggs will scramble if it boils." But in addition to that enzyme thing, if you don't boil it, the starch will not cook completely, and you'll wind up with a raw, starchy taste. Could that be the grit you're talking about?

Here are two recipes we're fond of, a traditional pastry cream and a higher-fat, microwave pastry cream. Neither is pasty or gritty.