Implements the obs-websocket 5.x challenge authentication hash.
Returns the authentication response for the given password, salt, and challenge. Callers must not log the returned value.