I owe a whole bunch of blog posts on more official Xiph topics, but tonight I present silliness.

I suppose it was unhealthy to even try to resist painting the newest server. Now off it goes to Oregon.

Coming up tomorrow, we have an IETF BoF on a proposed Working Group for FOSS audio codecs. It is one of the more... controversial BoFs in recent memory. Something of a main event... ( way more under the cut, and we'd like interested folks to lend their voice... )

With tonight's double-release (well... release candidates) of libvorbis 1.2.2 and libogg 1.1.4, this has been quite a month of software releases. In addition to Ogg and Vorbis, Ralph, Tim and Greg have cut a second alpha of Thusnelda and I managed to push out some vanity software on my own time-- a new release of gPlanarity and the beginnings of getting the Gimp monkey off my back. No, not the new resampler, but a number of TileCache fixes that looked pretty necessary.

Whoof. Getting tired. But cdparanoia, two more libvorbis releases and a whooole lotta documentation come next.

It's time for the latest Thusnelda encoder project update summary!

Although I haven't gotten much time to dive back into Thusnelda coding myself, Tim Terriberry, Greg Maxwell and others have continued the work along at a merry pace. In the past few weeks, they've finally replaced the leaky fDCT from the original VP3 and begun work on adjusting the main quantization matrices and, hopefully soon, adaptive quantization. These improvements all improve fine detail rendering and, somewhat unexpectedly, improve gradient rendering as well:

The screen caps above were produced by Theora 1.0 on the left and an experimental version of Thusnelda with early quant matrix optimization work in addition to the new fDCT on the right. Both clips were encoded in constant-quantizer mode and equal bitrates.

Other improvements, more details and the full update report here.

Today shall be the Day of Blogging in general. In this case, it even counts as much-needed documentation. There have been a couple of requests for a Theora/Thusnelda update since I have't posted one in really quite a while-- and that immediately after reporting 'rapid progress'.

Because I work on Xiph code as part of my day-job responsibilities at Red Hat, I don't have complete control over my development priorities. Red Hat in general is usually reasonably hands-off, but around Thanksgiving of last year transfered me to working exclusively on a high-priority internal project, and thus transfered me off all Xiph projects including Thusnelda. This is temporary, but temporary may be a few more months yet. Yeah, 'Grrrrr' is right.

The good news is that Xiph had already been working on securing funding to get Tim Terriberry working on Thusnelda full time when I got pulled off the project. Mozilla's Chris Blizzard and WikiMedia's Eric Moeller were especially keen to push development forward when we met at the Open Video Alliance's meeting in New Haven last October, and money for contracts came through in February (the $100K mentioned there is for several contracts, not just completing Thusnelda).

As much of the Thusnelda work was me channelling Tim, this is an even better position to be in than we were before. It would be best if we could both work on it, but Tim is definitely a higher-powered video hacker than I am.

Now... can I successfully rope Tim into blogging?

It's time for an update, especially during a period of rapid progress.

Over the past six months, I've been incrementally reorganizing the entire codebase from top to bottom. This in itself didn't really produce direct tangible performance benefits. However, I was finally able to implement the Holy Grail: efficient per-token RD optimization. This was the Big Win that would suddenly allow lots of big and small incremental improvements to pile up together and bring Theora's bitrate efficiency into the modern era.

Below is a short, sweet summary. Each image is a closeup of a frame, the encoded test clip is linked to the image:

Mainline Theora test clip: 580kbps

Current Thusnelda: 580kbps
Encoded with internal qi=50, skip lambda=5, token lambda=16

More details (and demo clips), as usual, on the demo page I prepared to keep Red Hat informed of my progress. The best part of all of this is that per-token opt was just the first big shoe to drop. The code reorg has all the ducks in a row. This should be quite a month for Theora.

The Thusnelda work demonstrates nicely the real liability of old bitrate formats: It's not that it isn't possible to update an old format like VP3 to modern performance, it's just that it takes more coding effort and cleverness to work within the structure of the old format than to cut your losses, break compatability and do it the newer easier faster way. Which is not to say I'm not looking forward to the day when we can finally extend Theora past the current spec again...

Cdparanoia 10pre2 is now available for download at xiph.org/paranoia. This release will be promoted to the final 10.0 release in two-weeks-ish assuming no new blockers. Package maintainers, now is the time to update your package subdirectories in SVN to reflect the upcoming 10.0 final release.

Cdparanoia 10pre2 includes the following changes over the previous perelease:

1) Substantial code review by Peter Creath factored in; this resulted in several one-off bugfixes that probably only affected code efficiency.

2) Fix a crash bug when falling off the end of the commad set test list.

3) Fix a crash bug when exiting out from a drive that claims to support CDDA but steadfastly refuses to return useful data.

4) Relicense cdparanoia to GPLv3; relicense underlying libs to LGPLv3.

5) Set DAP bit on MMC READ CD transfers; mostly eliminates the 'is the drive interpolating slagged samples or not?' inconsistency.

Happy hacking!

Note; I had originally thought Ian Hickson had written this quote, but I was mistaken. The point it makes is unchanged of course, but getting the attribution right is important.

Posted by Michael John Ventnor: "The distinction between the video format war and the browser war is that it is far more unlikely for the open video format to make a comeback. Firefox had to be completely immensely better than IE, coupled with 1.0 being hyped at the right time (when IE had security bulletins almost every day), and we're still battling hard. There is almost no compelling reason to choose Theora over WMV from a complacent person's perspective.

"I really hope we're not going to support the proprietary video formats in each respective backend, that would be an enormous mistake for the open web. You'd need to run a proprietary OS to view these proprietary formats on the web (without being sued), which is exactly the antithesis of the entire Mozilla project".

Posted by Hixie: "The Theora codec isn't unencumbered, despite popular opinion; it just hasn't had anyone claim patents on it publicly yet."

The same claim has been being made against Vorbis for ten years but those voices have finally mostly quieted and are being forgotten. Vorbis actually had public threats from Thomson (which were retracted, but the retractions were not widely reported).

Of course part of the problem is that there is no legal test for 'unencumbered'. Nor do you have to be infringing to get sued. We will always have a halting problem here. And yet, both Vorbis and Theora are ten years old with no claims against them. Would another ten be sufficient? Twenty?

Vague threats that come to nothing (nothing that is except eliminating the monopoly interest's competition) are the story of my professional life. Patent law is being used to extort outright, and most of the interests represented here are willingly buying into it.

This isn't even getting to the trolls. And we sit around and wonder why this industry is so dysfunctional:

Posted by Anonymous Coward: "The development of 'the odd one out' is controlled by a loose group of developers making changes to it in an uncontrolled manner."

Let me give names to this dark, shadowy, menacing 'loose group' here. The core project leads are myself (Monty), Ralph Giles and Timothy Terriberry, unchanged for the past five years. We're in control of the codec and format. I'm happy to stack this core against any codec team out there. 'Volunteer' does not make us 'uncontrolled'. What I assume Anonymous means here is that we're not controlled by him.

Posted by Anonymous Coward: "See the complaints on the theora mailing list about non-reviewed code being committed to their source code repository. Can you say 'patent nightmare'?"

I feel like I'm getting played by responding to claims like this, and yet they're repeated so often.

Here the anonymous troll is harping on a recent complaint involving committing an MMX/SSE acelleration patch that ran against one of Intel's implementation guidelines and we reverted. This had nothing to do with the format.

In other incarnations, the trolls latch onto messages sent by people watching the project who don't actually contribute code. I suppose "loose" and "uncontrolled" does indeed apply to some of our louder fanboys. I'll point out they don't submit (or commit) code.

But more to the point, we're a distributed organization and we have our technical discussions in public. Every minor bug, every minor misstep is there in plain view for the world to see. This is the very embodiment of 'transparency'. Apparently there are industry figures who don't comment in blogs using their real names suggesting that the programmers in their employ never make minor (or major) mistakes. Because they're committed in secret, or no one ever publically admits to errors, it's the same thing as mistakes never happening.

I really hope no one in any position of authority actually believes that.

1. Painfully detailed code review complete.
2. Original VP3 SAD pre-analysis determined to be extraneous.
3. The 64-pass-per-frame (no, I'm not kidding) DCT token packing loop replaced.
4. Multi-step, multi-multi-multi-pass coding tightened up to a pass-and-a-half loop.
5. New mode selection code based on theora-exp.
6. New motion estimation and compensation code.

The primary intent of the MV-specific work was to increase speed without impairing coding efficiency. This we've done to the tune of 1.5x improvement (Theora best case vs. Thusnelda) to 10x (Theora worst case vs. Thusnelda); additional planned work may yet double performace again.

Optimization of block type selection (mode select) was to increase coding efficiency. The new MV and mode selection code alone, compared directly to the algorithm used by mainline Theora, currently improves bitrate coding efficiency approximately 10%.

Above: example of mode selection in a frame of video from mainline Theora encode.

Mainline Theora as above, but directed to code all blocks and disable ZeroBin, such that the only differences we're comparing between mainline Theora and Thusnelda are the motion and mode code.

Above: example of mode selection in a frame of video from the Thusnelda encoder.

This is still brand new code, and we know of a few nasty bugs still in the new encoder-- just in case you should decide to play with the code and run into them. See the full page for the video including the above frames, including versions that include the debugging telemetry as part of the encoded video.

More on this subject later, but the pic is neat in its own right. Ongoing work debugging mode selection and the new motion vector engine in Thusnelda...

The Thusnelda decoder now supports visual debugging output, enabled via theora_control() calls. A small mplayer patch makes this available through mplayer via -theoradopts vismbmode=[number]:vismv=[number] where [number] is a bitmask of the modes to display. The image was generated using 0xdc, the block types with explicit or implicit motion vectors.

Today, I will entertain you with a solidly less professional sort of rant. Welcome to one of my hotbuttons.

From our gullible friends at Linux.com:

"Matroska's creators assert its superiority over Ogg in a number of areas; the one distinction no one argues about is that Matroska is designed to encapsulate any codec, while Ogg is designed specifically for Xiph.org codecs like Theora."

No. No, no, no. Dear God No. Not this again.

Ogg was not designed for only our codecs. Ogg will happily swallow data from any codec you like. We simply didn't hand the world the magic tools for doing it. Why? Because our mandate isn't promoting all codecs. It's promoting Free codecs. So, early on, we made an intentional decision to officially mostly ignore the existence of any encumbered codec. Don't acknowledge the non-Free, don't talk about the non-Free, don't code the non-Free. Correct or not in retrospect, the decision was to publically stay on topic: FOSS only. FOSS is the way.

So, we only mention the Free codecs in the docs and only implement the Free codecs in our own tools. As an aside, it's not entirely clear that it's even legal for us to implement, eg, MPEG4 support in the Ogg tools we distribute, and unlike a number of other FOSS codec organizations we don't winkingly dance back and forth across the legal lines regarding IP and patents ("You can't sue us! What we're doing isn't illegal in all countries! Hey, leggo! This is bullshit! We didn't do anything!")

There's nothing morally wrong with standing up to The Man, of course, especially when He's being a Real Prick. However, our organizational goal is to coopt The Man by showing Him how [E|e]verybody makes more money in the end the FOSS way. Because The Man is more likely to listen when you're not urinating on His Lawn, we avoid the pseudo-legal tapdancing that only works if you're too small to be worth squashing. But I digress.

Getting back to non-free codecs in Ogg, recall the old Dilbert cartoon where the pointy-haired boss proudly proclaims, "They said it couldn't be done!" Dilbert corrects him, "No, we said 'Don't do it.'" That's a big difference, and spin has no place in engineering.

To be fair, that's not directly parallel to the situation here. We're not actually saying 'Do not use Ogg with proprietary codecs or in closed applications' we're saying 'please go ahead, but we can't directly contribute; it's not our mandate.' Our non-profit charter, on file with the cheerful folks at the IRS, states that we spend our resources on the Free stuff. We're happy to see a rising tide lift all boats, closed and open, commercial and public. However, our internal mandate is the open and the public interests. For that reason we've only coded for the Free codecs, most of which happen to be ours(*).

(*)The interesting part is that it didn't start out that way. Speex and FLAC joined Xiph.Org later on. Dirac is a project of the BBC.

</rant>

Oh, and there's nothing wrong with Matroska. It's a good non-linear container counterpart to Ogg for doing many of the things Ogg was not designed to do. It's just that 'contain arbitrary codecs' is not on that list. 'Ogg can only contain Xiph codecs' is a common 'fact' brought up in the context of 'why company Foo won't support Ogg', so let's put a rest to this bit of nonsense, OK? It's harmful to the larger goals of FOSS.

Backstory: Theora is often attacked for having relatively low encoded image quality. The Doom9 2005 multi-codec shootout is the reference most commonly cited by detractors. I'm not going to criticize the Doom9 shootout because I think it was entirely fair and showed Theora legitimately falling short. I wrote http://web.mit.edu/xiphmont/Public/theora/demo.html long before the W3C/Theora discussion recently stole the limelight. Although I originally intended that document for internal consumption, it escaped into the public and is a useful look at how we at Xiph view Theora internally.

There's an important distinction to be made between the incapabilities of the Theora format, and the quality of the Theora encoder. Of course, users don't really care about theoretical performance speculation. To them, the software either works well or it doesn't. It doesn't matter if that's due to design or implementation. Sucking is sucking and it's hard to dispute that the Theora encoder, as inherited from On2, is downright sorry by modern standards. Believe me-- we're more intimately familiar with the level of suck than most.

However, if the software sucks but the format design does not, the software can be improved. That's why we've been working on the Thusnelda encoder for the past few months.

Another problem that comes up regularly is one of perception, and it's very simple. Theora is built from VP3 (true). The current generation of On2 codec is VP6/VP7 (true). '3' is a smaller number than '6' or '7' (true). It is half as big or less in fact (true)! VP6/7 must be about twice as advanced as VP3 (uhhhh). Thus, why would anyone use an outdated throwback like Theora, which is really VP3, which is less than half as good as a current codec (huh, wait, what)?

It's stunning how pervasive this argument actually is. For that reason, we've decided in the next release to rename 'Theora' to 'TV8'. Juuuust kidding.

First off, MPEG2 video, MPEG4 video, VP3, VP5 and VP6 are all in the same codec family. They're all block-DCT codecs with motion-compensated inter-frame block prediction. Exactly how they lay the bits into the stream differs, but the foundation math is all the same. They all use the same battle strategy. In fact, if you generalize 'DCT' to 'transform', even Dirac and Snow are typical members of this video codec family. (Tarkin differed significantly and was not a technical success).

Digging down lower, you begin to see differences in how the strategy is implemented tactically. Examples: At the last, lowest level entropy coding stage, VP3 uses a Huffman coding and VP6 uses range coding. VP3 uses Hilbert-ordering for blocks and macroblocks. VP6 decided the extra complexity of Hilbert ordering isn't worth the gain and just uses raster order. In both cases, they're doing the same thing-- but doing it in a slightly different way.

In short, VP6/7 is more an exercise in altering details without needing to worry about backwards compatibility than it is an invalidation of VP3. We've done and are doing the same thing with Theora, except that we're maintaining backward compatibility as we move forward. Of course, that glosses over several metric tons of details (and that's why we're not done yet). We have a finite details-per-hour throughput.

Note that we're maintaining compatibility across Theora versions; even the original Theora alphas had several changes that were incompatible with VP3, but the encoder could be directed to output in a VP3 compatibility mode. Our Thusnelda encoder no longer offers VP3 compatible output).

I've just gotten back from FOMS/LCA and then ten days in NZ. Immediately following two FOSS conferences with 'no net access whatsoever' was perhaps not the brightest possible plan. A number of things have happened in the past few weeks, both just before and during the trip. The biggest are non-technical: Patents. Patents. Patents.

The whole W3C HTML5 draft and Ogg brouhaha in December and January has made it clear in that our old strategy regarding patents regularly gets us utterly killed in the courts of mindshare and public opinion. "Say nothing, lest you accidentally incur liability", the lawyers have always said. So what do we do when a loose cannon in Nokia makes a ridiculous public statement like "MPEG4 is the open safe technology. Ogg is proprietary and a patent risk"? Obviously we have to be able to respond, and we have to be proactive enough to be prepared to respond.

The entire FOSS world is grappling with patent FUD. Fortunately, the legal thinking regarding patents is beginning to change now that legal minds are beginning to enter, understand and embrace the FOSS way of thinking. Transparency in the patent world is practically non-existent, but a few pioneering legal minds are finally coming to believe that lack of transparency is in fact the root of the problem. Once we were told, "Research nothing, know nothing, say nothing." The simple fact is, we've always researched and weren't about to stop. Now, we're being urged from multiple directions, "Publish everything you know. Document every challenge and rebuttal publicly." I'm ecstatic, even if this means a lot of work. Finally we can do something and do it in the light of day.

Right now I don't have more details to offer. However, I do have another good bit of news. Eben Moglen is one of the early explorers on the FOSS legal trail and his Software Freedom Law Center project has agreed to take us (the Xiph.Org Foundation) on as a client. Hopefully they're able to guide my idealism into something legally sound.

We'll have press releases about both the SFLC and the patent documentation project as soon as we can.