Several governments have tested formal index-based insurance to build climate resilience among smallholder farmers. Yet, adoption of such programs has generated concerns that insurance may crowd out long-established informal risk transfer arrangements. Understanding this phenomenon requires new analytic approaches that capture dynamics of human social behaviour when facing risky events. Here, we develop a modelling framework, based on evolutionary game theory and empirical data from Nepal and Ethiopia, to demonstrate that insurance may introduce a new social dilemma in farmer risk management strategies. We find that while socially optimal risk management is achieved when all farmers pursue a combination of formal and informal risk transfer, a community of self-interested agents is unable to maintain this coexistence at moderate to high covariate risks. We find that a combination of pro-social preferences - namely, moderate altruism and solidarity - helps farmers overcome these concerns and achieve the social optimum. Behavioural interventions that cue such preferences can render financial incentives more efficient in promoting optimal climate risk management, with potential savings worth approximately 5-15 percent of community agricultural income under a range of risk levels.