The easiest way to do that in Mixxx is to use the EQ kill buttons: kill the Mid to get mostly instrumental since vocals are in that range, or kill the high and low to get mostly vocals. For finer control, enable high-quality EQs in the preferences (un-check Static EQs) and tune the frequency cutoffs to your liking. If that's not enough for you, you'll need to use an external program that does advanced vocal-removal processing.
Mixxx Developer - Controller Specialist
Mixxx Windows package maintainer