Study of high-multiplicity three-prong and five-prong τ decays at BABAR