In negative hydrogen ion sources in situ adsorption of Cs is typically used to generate low work function converter surfaces. The achievement of a temporally stable low work function coating is, however, challenging due to the hydrogen plasma interaction with the surface. Particularly in ion sources for neutral beam injection systems for fusion with pulse lengths of minutes to hours temporal instabilities are a major issue and limit the source performance. To clarify the influence of the hydrogen plasma on the converter surface, investigations are performed at an experiment equipped with an absolute work function diagnostic based on the photoelectric effect. Caesiated surfaces are exposed to the full plasma impact by the generation of plasmas in front of the surface as well as to selected plasma species (H atoms, positive ions and VUV/UV photons) from an external plasma source to identify driving mechanisms that lead to surface changes. Depending on the exposure time and initial surface condition, the plasma strongly affects the surface in terms of work function and quantum efficiency (QE). For degraded Cs layers (work function >3 eV) a favorable increase in QE and reduction in work function can be achieved, while for Cs layers with an ultra-low work function of 1.2-1.3 eV the opposite is true. It is found that each plasma species can influence the Cs layers and that VUV photons lead to a work function increase of ultra-low work function layers. For sufficiently high VUV fluences a severe work function increase by 0.5 eV is given, highlighting the relevance of photochemical processes in the plasma-surface interaction and demonstrating that ultra-low work function layers are not stable in a hydrogen plasma environment.