Direct and inverse problems of mode scattering in rough-surfaced waveguides