Gravitational decoherence for mesoscopic systems