Did you install the entire fuel pump assembly, or just the fuel pump?
There is a check valve in the regulator and when that goes bad, you can have a bleed down of fuel pressure and the exact symptoms you are describing.
And if the engine stumbles and runs rough for a few seconds after it starts, you could have a leaky fuel injector. That leak bleeds down pressure and then it runs rough for a short time as the excess raw fuel is burned off.
Get a fuel pressure gauge and start testing your way to the root cause here.
Without data, all you have is an OPINION!
Past Jeeps: '49 ******, '81 Scrambler, '88 Comanche
Current Jeep: 1999 XJ Sport, bone stock, daily driver.
I work on Jeeps!!