A realistic model for the free rho meson with coupling to two-pion states is employed to calculate the rho propagator in a hot and dense hadron gas. The medium modifications are based on hadronic rescattering processes: intermediate two-pion states are renormalized through interactions with surrounding nucleons and deltas, and rho meson scattering is considered off nucleons, deltas, pions and kaons. Constraints from gauge invariance as well as the full off-shell dynamics of the interactions are accounted for. Within the vector dominance model we apply the resulting in-medium rho spectral function to compute $e^+e^-$ production rates from $\pi^+\pi^-$ annihilation. The calculation of corresponding $e^+e^-$ spectra as recently measured in central collisions of heavy-ions at CERN/SpS energies gives reasonable agreement with the experimental data.