Question in the title
The lips are not a muscle.
Yeah, but only if they are being used improperly. If that is the case then just keep them relaxed and together, otherwise their size is just genetic.
Sometimes you must relax the mentalis muscle. Other times the orbicularis oris. Sometimes you may need to engage the zygomaticus. Unless we know the exact cause your lip position and why they are tensed or enlarged is only speculation.
I know that proper oral posture made my lips look more aesthetic. I had two teeth extracted on the lower arch so lightly jutting my jaw gives me a natural lip fullness that looks appropriate for a male. But if I close my mouth with my narrow lower arch sitting deep inside the upper, my upper lip pouts too much and makes me look feminine.
So lip size is not as "genetic" as people make it out to be . For example bimax isn't genetic but it causes the lips to protrude excessively. Do you have bimax, OP?