Conventional remote speaker verification systems rely on the system to have access to the user’s recordings, or features derived from them, and also a model of the user’s voice. In the proposed approach, the system has access to none of them. The features extracted from the user’s recordings are transformed to bit strings in a way that allows the computation of approximate distances, instead of exact ones. The key to the transformation uses a hashing scheme known as Secure Binary Embeddings. An SVM classifier with a modified kernel operates on the hashes. This allows speaker verification to be performed without exposing speaker data. Experiments showed that the secure system yielded similar results as its non-private counterpart. This approach may be extended to other types of biometric authentication.