That's a couple of features at play.
First, if there's only one attached image the system will show it inline - right in the message rather than as an attachment at the bottom.
The setting allows for Disabled, Yes if only one attachment, or Yes (which means all attachments will be shown inline in the post.)
The other thing is what is called a light box. It's intended for quick display and flipping through of multiple images and is pretty darned slick if you ask me.
I don't see how it's any different than a new window popping up - you still need to close it, just like a new window. As boot said, there is a close button, or you can just click outside the picture, on the forum page and the lightbox will close.
The nice thing is that it allows you to flip through the images. you can always right click and save as if you're wanting to download them, or right click and Open In New Window if you're wanting to have multiple images open for comparison or whatever..
The inconsistency may be with the fact that if it's only one image, it doesn't use the light box. I can see that. I'll turn that off.