The invariant mass or rest mass of a relativistic system is the mass that is assigned to the system as a whole in the center-of-momentum frame, where the total momentum is zero. In that case, the mass-energy equivalence applies and we can state
where is the total relativistic energy of the system in the center-of-momentum frame (its rest energy). In frames with nonzero four-momentum, the invariant mass obeys
where is the four-momentum. This makes it closely related to the center-of-mass energy. Sometimes when using natural units is called the invariant mass and not . This is because in natural units so . In this context, it can be seen as the square of the the center-of-mass energy.
The invariant mass is not the mass of any particular object. Rather, it's a property of the system that naturally appears due to the mass-energy equivalence in resting frames. As the name suggests, it is a relativistic invariant. In many cases, it is also conserved. Specifically, if the system is isolated, energy and momentum are both conserved, so invariant mass is too. It is also conserved in all decays and elastic scatterings. It is not conserved in inelastic scatterings.
Rest mass is especially useful in particle physics, as it provides an invariant quantity that stays the same both before and after an elastic particle scattering or particle decay event. This is particularly useful in studying the kinematics of these events as, alongside conservation of momentum, it provides a universal statement that is quite useful in solving the equations, even in processes where the particles change.