We study the medium-induced energy loss $-\Delta E_0(L_p)$ suffered by a heavy quark produced at initial time in a quark-gluon plasma, and escaping the plasma after travelling the distance $L_p$. The heavy quark is treated classically, and within the same framework $-\Delta E_0(L_p)$ consistently includes: the loss from standard collisional processes, initial bremsstrahlung due to the sudden acceleration of the quark, and transition radiation. The radiative loss induced by rescatterings $-\Delta E_{rad}(L_p)$ is not included in our study. For a ultrarelativistic heavy quark with momentum $p \gsim 10 \ {\rm GeV}$, and for a finite plasma with $L_p \lsim 10 \ {\rm fm}$, the loss $-\Delta E_0(L_p)$ is strongly suppressed compared to the standard stationary regime $-\Delta E_0(L_p) \simeq -\Delta E_{coll}(L_p) \propto L_p$ valid when $L_p$ becomes large. This suppression is a consequence of the retardation affecting collisional processes - the latter cannot start at initial time when the heavy quark has not built its asymptotic proper field yet. Our results indicate that $-\Delta E_{rad}$ should be the dominant contribution to the total heavy quark energy loss, $-\Delta E_{tot} = -\Delta E_0 -\Delta E_{rad} \simeq -\Delta E_{rad}$, as indeed assumed in most of jet-quenching analyses. However they might raise some question concerning the RHIC data on large $p_{\perp}$ electron spectra.