I don't know of any quantitative measure, but I also haven't looked very hard.
There'd probably be something in Bunge's Texture Analysis (
https://www.researchgate.net/publication/277308168_Texture_Analysis_in_Materials_Science_H-J_Bunge)
Just having a quick look at things, eqn 4.237 imposes the constraint that the sums of the squares of the Cij values should be a minimum, but I don't know if this holds generally, as he is talking about extending l past l_max, whatever that may mean. That may help in preventing overfitting, but doesn't really define it.
http://www.ccp14.ac.uk/solution/gsas/spher_harmonics.html says that if the texture index (see eq 4.212) is greater than 3, then you're overfitting. But no quantitative statement.