On pavement at highway speeds, you have a LOT more body roll. If you daily-drive your Xterra and put a lot of highway miles on it, I would not take it off. It takes away your ability to swerve hard on the highway really, even after getting used to the no-swaybar dynamics.
I disagree. I dd my truck and find myself at highway speed regularly and drive the truck just as aggressively now as I did before I took it off. I did take me a LONG time to get used to not have a front sway bar on though. There's more roll taking sharp turns and weaving through highway traffic. Once you've trained yourself to anticipate it, you stop noticing it altogether.
To the OP, It's not a permanent decision. You can always take it off, store it in your garage or a closet and put it back on if you don't like it.