The Rabin cryptosystem is a family of public-key encryption schemes based on a trapdoor function whose security, like that of RSA, is related to the difficulty of integer factorization.^[1][2]

The Rabin trapdoor function has the advantage that inverting it has been mathematically proven to be as hard as factoring integers, while there is no such proof known for the RSA trapdoor function. It has the disadvantage that each output of the Rabin function can be generated by any of four possible inputs; if each output is a ciphertext, extra complexity is required on decryption to identify which of the four possible inputs was the true plaintext. Naive attempts to work around this often either enable a chosen-ciphertext attack to recover the secret key or, by encoding redundancy in the plaintext space, invalidate the proof of security relative to factoring.^[1]

Public-key encryption schemes based on the Rabin trapdoor function are used mainly for examples in textbooks. In contrast, RSA is the basis of standard public-key encryption schemes such as RSAES-PKCS1-v1_5 and RSAES-OAEP that are used widely in practice.

The Rabin trapdoor function was first published as part of the Rabin signature scheme in 1978 by Michael O. Rabin.^[3][4][5] The Rabin signature scheme was the first digital signature scheme where forging a signature could be proven to be as hard as factoring.

The trapdoor function was later repurposed in textbooks as an example of a public-key encryption scheme,^[6][7][1] which came to be known as the Rabin cryptosystem even though Rabin never published it as an encryption scheme.

Like all asymmetric cryptosystems, the Rabin system uses a key pair: a public key for encryption and a private key for decryption. The public key is published for anyone to use, while the private key remains known only to the recipient of the message.

The keys for the Rabin cryptosystem are generated as follows:

1. Choose two large distinct prime numbers ${\displaystyle p}$ and ${\displaystyle q}$ such that ${\displaystyle p\equiv 3{\bmod {4}}}$ and ${\displaystyle q\equiv 3{\bmod {4}}}$.
2. Compute ${\displaystyle n=pq}$.

Then ${\displaystyle n}$ is the public key and the pair ${\displaystyle (p,q)}$ is the private key.

A message ${\displaystyle M}$ can be encrypted by first converting it to a number ${\displaystyle m<n}$ using a reversible mapping, then computing ${\displaystyle c=m^{2}{\bmod {n}}}$. The ciphertext is ${\displaystyle c}$.

The message ${\displaystyle m}$ can be recovered from the ciphertext ${\displaystyle c}$ by taking its square root modulo ${\displaystyle n}$ as follows.

1. Compute the square root of ${\displaystyle c}$ modulo ${\displaystyle p}$ and ${\displaystyle q}$ using these formulas:

   ${\displaystyle {\begin{aligned}m_{p}&=c^{{\frac {1}{4}}(p+1)}{\bmod {p}}\\m_{q}&=c^{{\frac {1}{4}}(q+1)}{\bmod {q}}\end{aligned}}}$

2. Use the extended Euclidean algorithm to find ${\displaystyle y_{p}}$ and ${\displaystyle y_{q}}$ such that ${\displaystyle y_{p}\cdot p+y_{q}\cdot q=1}$.
3. Use the Chinese remainder theorem to find the four square roots of ${\displaystyle c}$ modulo ${\displaystyle n}$:

   ${\displaystyle {\begin{aligned}r_{1}&=\left(y_{p}\cdot p\cdot m_{q}+y_{q}\cdot q\cdot m_{p}\right){\bmod {n}}\\r_{2}&=n-r_{1}\\r_{3}&=\left(y_{p}\cdot p\cdot m_{q}-y_{q}\cdot q\cdot m_{p}\right){\bmod {n}}\\r_{4}&=n-r_{3}\end{aligned}}}$

One of these four values is the original plaintext ${\displaystyle m}$, although which of the four is the correct one cannot be determined without additional information.

We can show that the formulas in step 1 above actually produce the square roots of ${\displaystyle c}$ as follows. For the first formula, we want to prove that ${\displaystyle m_{p}^{2}\equiv c{\bmod {p}}}$. Since ${\displaystyle p\equiv 3{\bmod {4}},}$ the exponent ${\textstyle {\frac {1}{4}}(p+1)}$ is an integer. The proof is trivial if ${\displaystyle c\equiv 0{\bmod {p}}}$, so we may assume that ${\displaystyle p}$ does not divide ${\displaystyle c}$. Note that ${\displaystyle c\equiv m^{2}{\bmod {pq}}}$ implies that ${\displaystyle c\equiv m^{2}{\bmod {p}}}$, so c is a quadratic residue modulo ${\displaystyle p}$. Then

${\displaystyle m_{p}^{2}\equiv c^{{\frac {1}{2}}(p+1)}\equiv c\cdot c^{{\frac {1}{2}}(p-1)}\equiv c\cdot 1\mod p}$

The last step is justified by Euler's criterion.

As an example, take ${\displaystyle p=7}$ and ${\displaystyle q=11}$, then ${\displaystyle n=77}$. Take ${\displaystyle m=20}$ as our plaintext. The ciphertext is thus ${\displaystyle c=m^{2}{\bmod {n}}=400{\bmod {77}}=15}$.

Decryption proceeds as follows:

1. Compute ${\displaystyle m_{p}=c^{{\frac {1}{4}}(p+1)}{\bmod {p}}=15^{2}{\bmod {7}}=1}$ and ${\displaystyle m_{q}=c^{{\frac {1}{4}}(q+1)}{\bmod {q}}=15^{3}{\bmod {11}}=9}$.
2. Use the extended Euclidean algorithm to compute ${\displaystyle y_{p}=-3}$ and ${\displaystyle y_{q}=2}$. We can confirm that ${\displaystyle y_{p}\cdot p+y_{q}\cdot q=(-3\cdot 7)+(2\cdot 11)=1}$.
3. Compute the four plaintext candidates:

   ${\displaystyle {\begin{aligned}r_{1}&=(-3\cdot 7\cdot 9+2\cdot 11\cdot 1){\bmod {77}}=64\\r_{2}&=77-64=13\\r_{3}&=(-3\cdot 7\cdot 9-2\cdot 11\cdot 1){\bmod {77}}=\mathbf {20} \\r_{4}&=77-20=57\end{aligned}}}$

and we see that ${\displaystyle r_{3}}$ is the desired plaintext. Note that all four candidates are square roots of 15 mod 77. That is, for each candidate, ${\displaystyle r_{i}^{2}{\bmod {77}}=15}$, so each ${\displaystyle r_{i}}$ encrypts to the same value, 15.

The Rabin cryptosystem can be used to create and verify digital signatures. Creating a signature requires the private key ${\displaystyle (p,q)}$. Verifying a signature requires the public key ${\displaystyle n}$.

A message ${\displaystyle m}$ can be signed with a private key ${\displaystyle (p,q)}$ as follows.

1. Generate a random value ${\displaystyle u}$.
2. Use a cryptographic hash function ${\displaystyle H}$ to compute ${\displaystyle c=H(m\mathbin {\Vert } u)}$, where the double-bar denotes concatenation. ${\displaystyle c}$ should be an integer less than ${\displaystyle n}$.
3. Find a square root of ${\displaystyle c}$ using the private key ${\displaystyle (p,q)}$. This will produce the usual four results, ${\displaystyle r_{1},r_{2},r_{3},r_{4}}$.
4. One might expect that squaring each ${\displaystyle r_{i}}$ would produce ${\displaystyle c}$. However, this will be true only if ${\displaystyle c}$ happens to be a quadratic residue modulo ${\displaystyle p}$ and ${\displaystyle q}$. To determine if this is the case, square ${\displaystyle r_{1}}$. If this does not yield ${\displaystyle c}$, repeat this algorithm with a new random ${\displaystyle u}$. The expected number of times this algorithm needs to be repeated before finding a suitable ${\displaystyle c}$ is 4.
5. Having found a square root ${\displaystyle r_{1}}$ of ${\displaystyle c}$, the signature is ${\displaystyle (r_{1},u)}$.

A signature ${\displaystyle (r,u)}$ for a message ${\displaystyle m}$ can be verified using the public key ${\displaystyle n}$ as follows.

1. Compute ${\displaystyle c=H(m\mathbin {\Vert } u)}$.
2. Compute ${\displaystyle r'=c^{2}{\bmod {n}}}$
3. The signature is valid if ${\displaystyle r'\mathrel {\stackrel {?}{=}} r}$ and a forgery otherwise.

Decrypting produces three false results in addition to the correct one, so that the correct result must be guessed. This is the major disadvantage of the Rabin cryptosystem and one of the factors which have prevented it from finding widespread practical use.

If the plaintext is intended to represent a text message, guessing is not difficult; however, if the plaintext is intended to represent a numerical value, this issue becomes a problem that must be resolved by some kind of disambiguation scheme. It is possible to choose plaintexts with special structures, or to add padding, to eliminate this problem. A way of removing the ambiguity of inversion was suggested by Blum and Williams: the two primes used are restricted to primes congruent to 3 modulo 4 and the domain of the squaring is restricted to the set of quadratic residues. These restrictions make the squaring function into a trapdoor permutation, eliminating the ambiguity.^[8]

For encryption, a square modulo n must be calculated. This is more efficient than RSA, which requires the calculation of at least a cube.

For decryption, the Chinese remainder theorem is applied, along with two modular exponentiations. Here the efficiency is comparable to RSA.

It has been proven that any algorithm which finds one of the possible plaintexts for every Rabin-encrypted ciphertext can be used to factor the modulus ${\displaystyle n}$. Thus, Rabin decryption for random plaintext is at least as hard as the integer factorization problem, something that has not been proven for RSA. It is generally believed that there is no polynomial-time algorithm for factoring, which implies that there is no efficient algorithm for decrypting a random Rabin-encrypted value without the private key ${\displaystyle (p,q)}$.

The Rabin cryptosystem does not provide indistinguishability against chosen plaintext attacks since the process of encryption is deterministic. An adversary, given a ciphertext and a candidate message, can easily determine whether or not the ciphertext encodes the candidate message (by simply checking whether encrypting the candidate message yields the given ciphertext).

The Rabin cryptosystem is insecure against a chosen ciphertext attack (even when challenge messages are chosen uniformly at random from the message space).^[6]: 214 By adding redundancies, for example, the repetition of the last 64 bits, the system can be made to produce a single root. This thwarts this specific chosen-ciphertext attack, since the decryption algorithm then only produces the root that the attacker already knows. If this technique is applied, the proof of the equivalence with the factorization problem fails, so it is uncertain as of 2004 if this variant is secure. The Handbook of Applied Cryptography by Menezes, Oorschot and Vanstone considers this equivalence probable, however, as long as the finding of the roots remains a two-part process (1. roots ${\displaystyle {\bmod {p}}}$ and ${\displaystyle {\bmod {q}}}$ and 2. application of the Chinese remainder theorem).

• Topics in cryptography
• Blum Blum Shub
• Shanks–Tonelli algorithm
• Schmidt–Samoa cryptosystem
• Blum–Goldwasser cryptosystem
• Kunerth's algorithm

• Buchmann, Johannes. Einführung in die Kryptographie. Second Edition. Berlin: Springer, 2001. ISBN 3-540-41283-2
• Menezes, Alfred; van Oorschot, Paul C.; and Vanstone, Scott A. Handbook of Applied Cryptography. CRC Press, October 1996. ISBN 0-8493-8523-7
• Rabin, Michael. Digitalized Signatures and Public-Key Functions as Intractable as Factorization (in PDF). MIT Laboratory for Computer Science, January 1979.
• Scott Lindhurst, An analysis of Shank's algorithm for computing square roots in finite fields. in R Gupta and K S Williams, Proc 5th Conf Can Nr Theo Assoc, 1999, vol 19 CRM Proc & Lec Notes, AMS, Aug 1999.
• R Kumanduri and C Romero, Number Theory w/ Computer Applications, Alg 9.2.9, Prentice Hall, 1997. A probabilistic for square root of a quadratic residue modulo a prime.

• Menezes, Oorschot, Vanstone, Scott: Handbook of Applied Cryptography (free PDF downloads), see Chapter 8