Any neutral boson such as a dark photon or dark Higgs that is part of a non-standard sector of particles can mix with its standard model counterpart. When very weakly mixed with the Standard Model, these particles are produced in the early Universe via the freeze-in mechanism and subsequently decay back to standard model particles. In this work, we place constraints on such mediator decays by considering bounds from Big Bang nucleosynthesis and the cosmic microwave background radiation. We find both nucleosynthesis and CMB can constrain dark photons with a kinetic mixing parameter between log ǫ ∼-10 to-17 for masses between 1 MeV and 100 GeV. Similarly, the dark Higgs mixing angle ǫ with the Standard Model Higgs is constrained between log ǫ ∼-6 to-15. Dramatic improvement on the bounds from CMB spectral distortions can be achieved with proposed experiments such as PIXIE.