Hierarchical Learning in Multi-Layer Population Games

Classical population games model agent learning in large populations but assume that agents myopically update decisions within a single-layer environment. However, many real-world systems involve hierarchical decision-making and decisions at each layer often generate instantaneous rewards or costs. This article introduces multi-layer population games, incorporating intermediate rewards across hierarchy layers. This multi-layer multi-reward structure induces a dynamic game, unifying the population game and dynamic game frameworks. Since standard Nash equilibria are no longer adequate in this setting, we define a subgame perfect Nash equilibrium and establish its connection to classical equilibrium notions. We then prove that the trajectories of the hierarchical learning dynamics asymptotically approach the set of Nash equilibria, under conditions characterized using passivity-based concepts. Finally, we establish a one-to-one correspondence between multi-layer population games and Markov decision process (MDP)-type mean-field games. This correspondence, combined with our convergence results, enables the development of a learning-while-playing equilibrium-seeking algorithm that overcomes the limitation of requiring a fixed mean-field during policy updates.