The purpose of this paper is to apply a version of homotopy technique to excited nonlinear problems. A modulation for the homotopy perturbation is introduced in order to be successfully for nonlinear equations having periodic coefficients. The nonlinear damping Mathieu equation has been studied as a simplest example. The analysis proceeds without assuming weakly nonlinearity and without presence of small factor for the periodic term. In this analysis, two nonlinear solvability conditions are imposed. One of them imposed in the first-order homotopy perturbation and used to study the stability behavior at resonance and non-resonance cases. The second level of the perturbation produces another solvability condition and used to determine the unknowns appear in solution for the first-order solvability condition. The method can be, also, used for excited linear equation. Stability conditions, and also the transition curves, are formulated independent of the small parameter i.e. in the unperturbed form as an alternative to classical methods.
Keywords: Homotopy perturbation method; nonlinear Mathieu equation; modulation method; parametric resonance