I've basically only purchased manuals my entire life and started driving around 12 years old, both my xterra and frontier are manuals, as is our family '80 CJ7 as far as 4wd stuff goes.
With that said, I'm definitely in the "manuals are not ideal for technical offroading" group even though shifting is so second nature to me I don't even think about it anymore. Beyond that, unless you really mess with the gearing, I think they become a liability in many low gearing situations. Made the mistake of taking the CJ over black bear pass about 10 years back and that is something I will never do again under any circumstance and to me is the extreme epitome of why manuals suck for real off roading. For normal non crawling stuff, manuals are totally fine IME and way more fun to drive, but some water situations and stuff that needs really low gearing or where stalling or even braking can put you in major trouble, they're just not fun or safe in some situations. If I had a dedicated offroad rig to mess with gearing I might change my own mind but for non purpose vehicles I vastly prefer an automatic for a lot of stuff.