Multiple equilibria and oscillations in a predator-prey system: Dynamics from coupling prey group defense and predator competition

In this paper, we revisit a predator-prey system that incorporates both prey group defense and predator intra-specific competition. Our results show that the model can exhibit complicated bifurcations, including a cusp type degenerate Bogdanov-Takens bifurcation of codimension three and a degenerate Hopf bifurcation of codimension two as the parameters vary. The coexistence of three positive steady states, along with the existence of multiple periodic oscillations and homoclinic orbits, is also shown. Moreover, we identify two distinct long-term dynamic regimes based on key parameter thresholds: a coexistence regime ($ \gamma<\gamma_{1} $, or $ \gamma = \gamma_{1} $ and $ \delta\leq \delta_{1} $), characterized by multiple equilibria or sustained oscillations, and a predator extinction regime ($ \gamma>\gamma_{1} $, or $ \gamma = \gamma_{1} $ and $ \delta> \delta_{1} $), where elevated mortality or intense competition drives the predator population to extinction. These results highlight the dual effect of competition on predators: moderate intensity stabilizes coexistence; however, when coupled with strong group defense, excessive competition may drive them to extinction.