Well, they sure didn't make much use of them so why shouldn't we get them?

Actually, there were separate processes going on with neandertals developing in Europe and the near east under selective pressure from ice age weather while modern humans seem to have developed in Africa at about the same time. By the time the latter showed up in neandertal lands they were already advanced and advancing...

I've run across speculation that autism-spectrum traits derive from the survival of Neanderthal genes in the modern human population. Given that autistic males tend to have larger than average brains, this seems to support that conjecture.

I have heard that among our species, brain size and IQ have no correlation. Among different species, brain sizes and intelligence is usually linked though.

My sources for this? Just my vague recollections

I just looked it up, but it seems there is conflicting theories. Most seem to think the higher the Brain Mass to Body Mass ratio, the more intelligent a creature is.

Conflicting theories means that they don't know so the theorists' favorite beliefs determine the interpretation of the data -- fine till more definitive information come in -- or the reality is being obscured for reasons of several persons' pet positions.