[Bernardo Innocenti] did a quick and dirty analysis of Xorg LoC vs Linux LoC, to give a "X codebase is roughly 1/2 Linux codebase" (which surprised me, actually, I'd always thought it was larger) and then a similarly rough analysis that suggests that Xorg has roughly 1/20th of the developer base.
To some extent Bernardo's metrics are arguable, but the basic facts are that Linux gets the lions share of the developers. Is this what we should expect? Is this a healthy way for the free and open source software community to support the development of what I personally would like to see as the future desktop base?
There are a lot of factors involved in determining which projects get more attention. I'd imagine that things like the amount of domain knowledge required to contribute a simple patch, the structure of the code, the amount of press that a project and its maintainers get, and the presence (or not) of a charismatic and photogenic leader. You could probably even base your entire academic career on studying this question.
But I also think we shouldn't underestimate the impact of paid jobs. Admittedly, I have never asked whether I could get paid before jumping off a cliff and starting a new project. But on the other hand, I now effectively work two jobs and only get paid for one. Most people are not as stupid as me, and will rationally choose to spend their time on tasks that they actually get compensated for. Even people who don't care about being paid will find it difficult to continue contributing over time due to practical issues of their personal life, families, etc, and they won't be able to contribute as much as if they were working full-time.
It's well-known in the programming community that if you can hack the Linux kernel, you can pretty much write your own job description . On the other hand, I've never heard that companies are looking to hire skilled X programmers. A quick pair of searches on that font of knowledge, Google, seem to back this up. Compare the results for x programming jobs -"active x" -"OS X" and linux kernel programming jobs. Nothing on the first page has anything to do with being paid to work on X.org, whereas the second search gives me an entire page full of people offering to pay me money if only I knew how to hack the kernel. KernelTrap, a central site for information on the kernel, even has a job board just for jobs related to kernel programming.
That's not to say that the X coders don't get paid; I'm sure many of them are compensated for their work on X, and given the level of expertise involved they're probably paid well. Companies that manufacture graphics cards have an interest in employing people to hack on X; Linux distributions and embedded hardware manufacturers also benefit from a better X and strike me as likely places of employment for X hackers. But there doesn't seem to be anything for X.org like the cornucopia of available and well-advertised jobs related to the Linux kernel. Programmers who want to be paid someday will, on average, end up gravitating to the project where the prospects are obvious, not hidden.
 Note that for the purposes of this discussion, it doesn't matter whether this common wisdom is true. The perception of jobs available in Linux-land will tend to cause people to head towards Linux regardless of whether the jobs exist or not.