Ogg Vorbis is a good choice because the sound quality is among the best of the newest formats out there. Recent double-blind listening tests put Ogg Vorbis among the highest quality of all the "second-generation" compressed audio codecs. This means you either save space and get the same quality, get higher quality for the same space requirements, or a combination of both (i.e. a little smaller and a little better-sounding).
Secondly, Ogg Vorbis is not only Open Source (BSD license), but is completely patent-free. This means that hardware manufacturers wanting to support Ogg Vorbis in their portable music players can do so without paying license fees, unlike most other formats. Software developers can use the Ogg Vorbis format for music/sounds in their games without having to get permission from some powerful company and without paying royalties. And the open nature of the code for the format means that many people have the freedom to port the tools to many other systems or add features, fix bugs and improve the code if they so desire. In fact, the BSD license allows for developers to modify their code to suit their own needs, and they don't even have to publish their changes! Most other formats are heavily patented and tightly controlled.
Finally, the format is well-designed to have several features some of the others don't. Those familiar with id3 tags for mp3 files will be well aware of their limitations; Ogg Vorbis features a flexible tagging standard which allows complete customization of tags for a given file, including user-defined tags (like "remixed by" or whatever you like).
Ogg Vorbis files support "bitrate peeling", which means you can produce a lower bitrate file from a higher bitrate file without re-encoding and at the same quality as if you'd encoded the file directly into the lower bitrate from the original file. No other lossy audio codec currently supports this. (N.B. Current files are peelable, but not very well. Good peeling support requires the encoder to be redesigned to store data in a more peeler-friendly (but still backwards-compatible) format. This is being worked on, slowly, but isn't currently a high priority.)
And Ogg Vorbis files are not limited to merely two channels of audio (left and right). They support up to 255 distinct channels, and thus are a natural fit for encoding the 6 channels of DVD audio alongside your DivX ;-) video.
Just to be clear, strictly speaking, the name "Ogg" refers to a generic container format which could hold many types of multimedia files (lossy compressed audio (Ogg Vorbis), lossy compressed audio designed for speech (Ogg Speex), lossless compressed audio (Ogg Flac), lossy compressed video (Ogg Theora), etc). "Vorbis" is the lossy compressed audio codec which is typically transported in Ogg files. "Ogg Vorbis" refers to both parts together: an Ogg format file containing audio compressed using Vorbis. And throughout this document, I've used the term "ogg" to generically mean "a file containing compressed audio in Vorbis format" since that's the file's extension, just like I've used "mp3" to mean "a file containing compressed audio in MPEG layer 3 format".