i would say yes it will build up and be a problem, i would also say dump the ball and let the tea loose! use a strainer, you will get better tea as it has more room to move and develop, also just tap out the strainer over the compost pot on your worktop / or whereever yo have one while you work in the kitchen.