If I remove your sense of balance
arm
eyes
alertness
memories
toes
hair
adrenal gland
artistic ability
sex drive
, is that you? No.
To be sure, the you that is left is less, but it's still you.

It's like lego. Which piece of the lego man is his "self"?