Scientific test: mp3 vs. ogg

I wanted to do a test to settle this matter as far as I could.

My hypothesis:
The best mp3s and the best oggs will *both* sound great, and will have no
subjectively audible difference with the source wav file. In other words, at
comparable file size at reasonably high quality settings, the comparisons
between them are moot.

Method:
compare output of the 'lame' version 3.93MMX encoder (mp3) with that of
'oggenc' V1.0 (libvorbis 1.0) (ogg) for a 15 second source wav file of
harpsichord music. I realize that what we really are comparing is not just
two audio compression algorithms, but two implementations of those
algorithms. But what else could we do? So, yes, I'm comparing 'lame' to
'oggenc'.

Since it's hard to compare apples and oranges as per compression, I simply
used a very high-quality setting on lame's command line (if you're
interested, it was a preset "--preset extreme" which gives what is considered
better quality than what is normally demanded of even the most severely
critical listener). Then, I tried to find a quality setting on 'oggenc' which
was closest in output file size. It happened to end up being '-q 7' (quality
setting 7 in the choosable range of -1 to 10, showing that at least oggs can
go to higher quality if desired than can mp3s, at least on these programs,
but at the obvious expense of increased file size)

The file output by lame, "snippet.mp3", was 319K.
The file output by oggenc, "snippet.ogg", was 323K.
This was as close as I could make them in size to make any comparison.

The "snippet.mp3" had one noticeable artifact, a "crescendo" at 3 seconds in.
The rest of the file then stayed at an amplified level which was *not*
reflected in the source file. Whether this is a bug in 'lame' or a feature of
mp3 encoding I don't know.

"snippet.ogg" OTOH, was not at all noticeably different from the source,
"snippet.wav".

At least for now, until anyone has a better mp3 encoder on Linux for me to
try, or proof that mp3 algorithms are less bug-ridden, I'm casting my vote
for using 'oggenc' where fidelity is concerned.

If anyone wishes to conduct this test themselves, I'm putting the audio source
and output files up at akjmusic:

http://www.akjmusic.com/audio/snippet.wav
http://www.akjmusic.com/audio/snippet.mp3
http://www.akjmusic.com/audio/snippet.ogg

I'm interested in any other observations anyone might come up with.

Cheers,
--
Aaron Krister Johnson
http://www.akjmusic.com
http://www.dividebypi.com

>My hypothesis:
>The best mp3s and the best oggs will *both* sound great, and will have
>no subjectively audible difference with the source wav file. In other
>words, at comparable file size at reasonably high quality settings, the
>comparisons between them are moot.

Yes, both codecs should achieve transparency, but the question is:
at what bitrate?

Except for clams. It's usually possible to design a signal that
gets audibly messed up after encoding, even at the highest bitrates.
Sometimes such signals occur in real music. Ogg hasn't been tested
nearly as well as LAME at high bitrates.

>The file output by lame, "snippet.mp3", was 319K.
>The file output by oggenc, "snippet.ogg", was 323K.
>This was as close as I could make them in size to make any comparison.
>
>The "snippet.mp3" had one noticeable artifact, a "crescendo" at
>3 seconds in.

I don't hear it.

>At least for now, until anyone has a better mp3 encoder on Linux for
>me to try, or proof that mp3 algorithms are less bug-ridden, I'm
>casting my vote for using 'oggenc' where fidelity is concerned.

You can't base things on a single test.

-Carl

>>At least for now, until anyone has a better mp3 encoder on Linux
>>for me to try, or proof that mp3 algorithms are less bug-ridden,
>>I'm casting my vote for using 'oggenc' where fidelity is concerned.
>
>You can't base things on a single test.

And to really be worthy of the "scientific" appellation, you should
be able to AB-X it. Have you tried?

-Carl

On Wednesday 29 September 2004 07:21 pm, Carl Lumma wrote:
> >My hypothesis:
> >The best mp3s and the best oggs will *both* sound great, and will have
> >no subjectively audible difference with the source wav file. In other
> >words, at comparable file size at reasonably high quality settings, the
> >comparisons between them are moot.
>
> Yes, both codecs should achieve transparency, but the question is:
> at what bitrate?
>
> Except for clams. It's usually possible to design a signal that
> gets audibly messed up after encoding, even at the highest bitrates.
> Sometimes such signals occur in real music. Ogg hasn't been tested
> nearly as well as LAME at high bitrates.
>
> >The file output by lame, "snippet.mp3", was 319K.
> >The file output by oggenc, "snippet.ogg", was 323K.
> >This was as close as I could make them in size to make any comparison.
> >
> >The "snippet.mp3" had one noticeable artifact, a "crescendo" at
> >3 seconds in.
>
> I don't hear it.

It's there...at least when played back by 'xmms'....anyone else hear it. It's
quite obvious to me, but I realize it could just be a bug in the playback app
as well.

I'll try and get my wife to do an AB-X compare with me sometime.

Best,
Aaron Krister Johnson
http://www.akjmusic.com
http://www.dividebypi.com

>I'll try and get my wife to do an AB-X compare with me sometime.

There's software that'll do it for you (at least on Windows).

-Carl

--- In MakeMicroMusic@yahoogroups.com, Carl Lumma <ekin@l...> wrote:

> And to really be worthy of the "scientific" appellation, you should
> be able to AB-X it. Have you tried?

Here's an old article, on mostly on low bit rates, with emendations;
in the updated version ogg is claimed to be better than mp3:

http://ekei.com/audio/

Here's some groovy biostatistics; ogg and mpc come out winners:

http://ff123.net/128test/interim.html

The problem is finding up to date information, it would seem, but it
seems clear my subjective judgment that ogg sounds better at identical
achieved compressions than mp3 (which is why I picked it) is widely
shared.

>> And to really be worthy of the "scientific" appellation, you should
>> be able to AB-X it. Have you tried?
>
>Here's an old article, on mostly on low bit rates, with emendations;
>in the updated version ogg is claimed to be better than mp3:
>
>http://ekei.com/audio/
>
>Here's some groovy biostatistics; ogg and mpc come out winners:
>
>http://ff123.net/128test/interim.html
>
>The problem is finding up to date information, it would seem, but it
>seems clear my subjective judgment that ogg sounds better at identical
>achieved compressions than mp3 (which is why I picked it) is widely
>shared.

Does this have anything to do with anything I've said?

-Carl

--- In MakeMicroMusic@yahoogroups.com, Carl Lumma <ekin@l...> wrote:

> Does this have anything to do with anything I've said?

Your subject line said "Scientific test: mp3 vs. ogg". Was your
posting on some different topic?

>> Does this have anything to do with anything I've said?
>
>Your subject line said "Scientific test: mp3 vs. ogg". Was
>your posting on some different topic?

I've said probably a dozen times on these lists that vorbis
is better at low bitrates. You've said the same thing as many
times, and more than once already in this thread.

I asked Aaron if he'd tried AB-X on his sample. How does your
reply relate to that question, if at all? I've also asked you
two direct questions in this thread, so if you're anxious to
reply to something there's a great place to start.

-Carl