A unified reinforcement learning approach is proposed to solve non-cooperative (MFG) and cooperative games (MFC) with large populations of players. The numerical scheme is designed as a two time scale stochastic approximation. We show how its calibration depends strictly on the nature of the problem. Convergence analysis and numerical examples are discussed.