If you are not familiar with Thomas Nagel's paper What It's Like To Be a Bat, what I'm going to say may not make sense.
Frank Jackson is right in saying that Nagel speaks as if the problem he is raising is one of "extrapolating from knowledge of one experience to another, of imagining what an unfamiliar experience would be like on the basis of familiar ones." In terms of Hume’s example, from knowledge of some shades of blue we can work out what it would be like to see other shades of blue. Nagel argues that the trouble with bats (or Martians, squirrels, etc) is that they are too unlike us. But it is hard to see an objection to Physicalism here. Physicalism makes no special claims about the imaginative or extrapolative powers of human beings, and it is hard to see why it need do so.
But I think the Knowing How/Knowing That distinction breaks this argument apart. I am not sure who originally formulated this distinction, however. (It was probably in the 80s when all the Anglo-American analytic professors congregated in their philosophy department hallways to argue about Nagel, Kripke and Jackson et al.) The distinction is between knowing how it is that light interacts upon the faculty of vision and knowing how this is experienced to the mind whose vision is doing the perceiving. This is, in fact, the distinction Nagel makes, except with much more alleviating clarity and less mysterianism. Mary, the omniscient scientist with respect to physical properties, knows how it is that physical properties work to produce experiences of light and color upon my perceptive abilities. But she does not know how it is that the experience should feel to me.
Does this mean physicalism is false? Hardly. Everything that has happened in my mind is still a series of physical events, even the experience itself is explained in physical terms. But the particular way in which that should feel to me, or what-it-is-like to experience it, is not readily available to Mary the Scientist. Just as it is inconceivable to us that we would know everything Mary knows about physics, so it is inconceivable that Mary would know the experiences themselves, extrapolated from my inner experience. She knows everything, granted, about my physical happenings. But she does not experience what I am experiencing when I see red colors.
For dualists to say that she must also have the experiences I am having, in effect, if she knows everything about the physics of my mind, misunderstands the scope and reason of physicalism.