When finishing a paddle, I like to leave the handle with just the oil finish. After the initial coat, I like to wet sand subsequent coats with 320 (apply oil and then sand using the oil), then 400 on any other coats.
This will give a nice silky finish on the handle, that your hand will be able...