Particles count rates at given Earth location and altitude result from the convolution of (i) the interstellar (IS) cosmic-ray fluxes outside the solar cavity, (ii) the time-dependent modulation of IS into Top-of-Atmosphere (TOA) fluxes, (iii) the rigidity cut-off (or geomagnetic transmission function) and grammage at the counter location, (iv) the atmosphere response to incoming TOA cosmic rays (shower development), and (v) the counter response to the various particles/energies in the shower. Count rates from neutron monitors or muon counters are therefore a proxy to solar activity. In this paper, we review all ingredients, discuss how their uncertainties impact count rate calculations, and how they translate into variation/uncertainties on the level of solar modulation $\phi$ (in the simple Force-Field approximation). The main uncertainty for neutron monitors is related to the yield function. However, many other effects have a significant impact, at the 5-10% level on $\phi$ values. We find no clear ranking of the dominant effects, as some depend on the station position and/or the weather and/or the season. An abacus to translate any variation of count rates (for neutron and $\mu$ detectors) to a variation of the solar modulation $\phi$ is provided.