It's not actually speed that matters; it's (neglecting the effects of bow rise with speed) the angle between the boat's centerline and the direction of motion. In aerodynamics we call this the "relative wind".
Picture a canoe moving straight forward, and you're looking straight down at the boat from above. We'll consider 2D flow, ignoring the water flowing under the boat*. In this 2D view, half of the water flows around one side of the boat and half flows around the other side. Right on the bow is what's termed the "stagnation point", where the flow divides.
Now say you're sitting amidships and sculling. The canoe is moving straight sideways, and the stagnation point is at the middle of the hull.
In between, when the boat is sideslipping while moving forward, the stagnation point is somewhere in between, and it moves forward as the sideslip angle decreases.
When things are symmetrical (straight forward or straight sideways), the center of pressure (the pivot point, more or less) is in line with the stagnation point. In between, when things aren't symmetrical, the stagnation point and center of pressure don't directly coincide, but they move in a similar manner.
The point is that the actual speed doesn't matter; it's the angle: if you're sideslipping twice as fast while also moving forward twice as fast, the angle is the same and stagnation point and pivot point don't move.
*Yes, 3D flow complicates things, but the general principle still holds. And, of course, the now lifting with speed changes the waterline length, which also moves the center of pressure aft even while the decreasing slip angle moves it forward... there's a lot going on. Hydrodynamics is messy.
