Such team dynamics sounds dangerous and should be avoided... Will the following help?
Let a team member take up the Scrum Master role - whether its the BA, Tech Lead, Developer or Tester in the team. It can even be rotated among them. Scrum Masters are pigs then and are close to the project to know/coach/communicate well... This would help keep the Scrum teams small and effective!
For this to happen, the following needs to be true though:
- Team is matured to practice Scrum
- Project doesn't demand a separate Scrum Master due to risk, size etc.