first step - as always - should be to _ensure_ that the sound input is as expected:
1) That the sound input works at all - you may check that by simply recording some vinyl.
A possible pitfall regarding this could be failure to distinguish between "playback" and "record" controls, be it ALSA or some windows controls. As mixxx "listens" to the incoming sound of the control vinyl, you therefore need the check the "record" controls. "Playback" only controls what the soundcard directly emits (e.g. what YOU hear, not the computer - this can be a little confusing in conjunction with "line in"...)
2) Stereo. For vinyl control the inputs definitely need to be in stereo - which is why I wonder if your first said hw-configuration would work: some microphone inputs are just mono.
One more thing: As far as I remember do newer mixxx versions have some sort of visual control of incoming sound - at the configuration dialogs.