I think the idea was that you knew that it'sa test and that you'd die if you remove the hand, so it's less willpower and more reasoning over instinct/fear, at least in theory. You have to presume the box is at least tuned to different people's pain thresholds or whatever.

Also, the text pretty much says that Mohiam is doing it wrong more or less on purpose:

"Enough," the old woman muttered. "Kull wahad! No woman child ever withstood that much. I must've wanted you to fail."

If you give the benefit of the doubt that Herbert figured out the practicalities and wasn't going by rule of cool (which he absolutely was) the implication is that the person administering the test has some control of the itnensity and you're supposed to deal with some pain you're supposed to hold, not become convinced that your hand is a charred stump like Paul is.

That, and the movie verisons amp the whole thing up a lot, so it comes across a bit differently.