Format-preserving encryption (FPE) is a data masking technique that encrypts a value into a different value of the same type and length — a 16-digit card number becomes another 16-digit number, a 9-digit national ID becomes another 9-digit value — so the protected data still fits the schema, constraints, and application logic your lower environments expect. In test data management, you reach for FPE when a field has to stay format-valid and consistent across tables but must not carry its real value, and when you may need to reverse the transformation later with a key. It is standardized by NIST in SP 800-38G (modes FF1 and FF3-1) and built on the AES block cipher.
What format-preserving encryption is
Format-preserving encryption is encryption whose output matches the shape of its input: the same data type, the same length, and the same set of allowed characters. Feed it a numeric field and you get a number back; feed it a 16-character string and you get a 16-character string. That constraint is the entire point: the ciphertext looks like the plaintext, so it can sit in the same column and move through the same code paths without anything downstream noticing the substitution. That property is why FPE is treated as one of the core data masking techniques rather than as general-purpose encryption. It sits among the masking techniques teams reach for on structured data, distinguished by keeping a value usable while stripping out its real content.
The technique is standardized. NIST specifies FPE in SP 800-38G, which defines two modes — FF1 and FF3-1 — both built on the AES block cipher. A secret key drives the encryption, and alongside it FPE takes a second, non-secret input called a tweak. The tweak lets one key produce different mappings in different contexts — a different transformation per column or per table, for instance — so you can vary the output without managing a separate key for every field.
One more property matters in practice. The current form of the second mode is FF3-1: researchers published a practical attack on the original FF3, and NIST revised the mode to close it, so FF3-1 is the version to use and plain FF3 is superseded. FPE also operates over a defined character domain — the set of symbols it treats as valid input, sometimes called the radix, such as the ten decimal digits or the twenty-six letters. A transform built for decimal digits cannot process a value that contains letters, and that domain boundary becomes a real constraint the moment you point FPE at messy production columns.
How FPE compares to tokenization, hashing, and random masking
The property that separates these techniques is reversibility — whether you can recover the original value, and what it costs you to do so. FPE is true, reversible encryption: anyone holding the key can recover the original, and nothing else is required. Tokenization replaces the value with an unrelated token and records the mapping in a separate token vault, so the original is recoverable only by looking it up there, not through any cryptographic operation. Hashing is one-way — it produces a fixed-length digest that cannot be reversed at all, which is why it fits verification and matching but not any workflow that needs the value back. Random or synthetic replacement swaps the real value for an invented one; it is irreversible, and unless it is deliberately seeded it is not consistent either, so the same input can produce different outputs on different runs.
It also clarifies how masking and encryption differ. Masking is the broad category: any transformation that replaces a sensitive value with a safe stand-in. Encryption is one specific, reversible way to do that, and FPE is the encryption technique that also preserves the shape of the data — which is exactly what lets it serve as a masking technique in the first place. Encryption that produced a longer, differently typed ciphertext would protect the value but break the column.
The practical line comes down to what you actually need. Reach for FPE when you need format fidelity together with the option to reverse a value or keep it consistent across tables, and you would rather not stand up and operate a vault to do it. Prefer tokenization when your goal is to have no cryptographic recoverability inside the environment at all — the real value never enters the lower environment, only a token that points back to a vault held somewhere more tightly controlled.
| Technique | Reversible? | Preserves format & length? | Consistent (same in → same out)? | Needs a vault? | Typical fit |
|---|---|---|---|---|---|
| Format-preserving encryption | Yes, with the key | Yes | Yes, given the key and tweak | No | Format-sensitive fields you may need to reverse or match |
| Tokenization | Via a vault lookup | Optional (format-preserving tokens) | Yes | Yes | Removing recoverable values from the environment entirely |
| Hashing | No (one-way) | No (fixed-length digest) | Yes | No | Verification and matching, never recovery |
| Random / synthetic replacement | No | Optional | Only if seeded | No | Realistic values you never need to reverse |
Why preserving format and length matters in lower environments
The payoff of preserving format and length is that the systems downstream of the data never notice it changed. A lower environment is full of code and rules that assume a specific shape, and a same-type, same-length value satisfies all of them without special handling. Column type and length constraints are the obvious case: a VARCHAR(16) column or a NUMERIC field rejects a value of the wrong width or type, and a format-preserved replacement slots in cleanly where a longer token or a blanked string would not.
The same logic runs through the rest of the stack. NOT NULL and CHECK constraints, fixed-width file parsers, and app-layer format validation all keep passing because the masked value is still shape-valid. Some fields carry an internal checksum — the Luhn digit on a payment card number is the common example — and there, format preservation gets you most of the way, with the transform configured to keep the checksum valid so any validator that recomputes it still passes.
Contrast that with naive masking that changes the length or type — blanking a card number to a run of Xs, or swapping a value for a long opaque token. Now you either alter the schema just for the test environment or watch test flows break wherever a parser or a validator chokes on the new shape, exactly the kind of breakage that makes masked data unusable for realistic testing and QA. For the engineers who own lower environments, format-invalid test data is a quiet tax: defects that only reproduce against real-shaped data slip through, and time goes to chasing failures that are artifacts of the masking rather than bugs in the code.
Keeping FPE consistent: determinism and referential integrity
Determinism is what makes FPE useful across a whole relational schema rather than one column at a time: given the same key and tweak, the same input always encrypts to the same output. A customer ID that appears in a dozen tables therefore encrypts to the same masked value in every one of them — the property that deterministic masking depends on. That consistency is also what lets relationships survive the masking pass. If a primary-key column and every foreign-key column that references it are transformed the same way, the joins that hold the database together still resolve, and referential integrity is preserved rather than quietly broken.
Tonic Structural is a clear example of FPE applied this way in practice. Structural's primary-key generators use FPE to transform a primary key consistently, and Structural then automatically applies the same generator to every foreign-key column that references that key — so the relationships that make a database queryable still work in the masked copy, without you wiring up each foreign key by hand. Because the transform is deterministic, a given key masks to the same value everywhere it appears, and a join across the masked tables returns what it should.
One implementation detail matters before you rely on it. In how Structural applies FPE, each generator is bound to a specific character domain and uses FPE to keep output in the input's format — Structural uses FPE to preserve format, not to certify against a particular standard, and some generators do not guarantee that output length exactly matches input length. The practical takeaway is to match the generator to the column's actual domain — a numeric transform for a numeric identifier, and a different one for a column that mixes letters and symbols.
The Tonic Advantage: consistency that survives the join. Assign an FPE-backed generator to a primary key in Tonic Structural, and Structural propagates the same transformation to every foreign key that references it. The customer ID masked in the orders table matches the customer ID masked in the accounts table, so a query that joins them returns the same rows it would in production. The database comes out queryable, not merely protected — which is the difference between test data you can actually develop against and a masked copy that no longer behaves like the original.
When to use FPE in test data — and its limits
Reach for FPE when a field is both format-sensitive and regulated — payment card numbers, national IDs, bank account numbers — and when the masked value has to stay consistent across tables or be reversible later. In Tonic Structural, FPE is one of several configurable techniques you can apply per column — alongside masking, tokenization, generalization, and scrambling — so you can reserve it for the fields that need format fidelity and handle the rest differently.
The limits decide where FPE is the wrong choice. FPE needs a sufficiently large value domain to be secure — NIST specifies a minimum domain size — so a field with only a handful of possible values — a single status flag, a short code — cannot be meaningfully protected: there are too few possible outputs to hide the input among. FPE is also reversible by design, so anyone with the key can recover the original values, and key management becomes part of your threat model. It moves the risk onto the custody of the key rather than removing the sensitive data from the environment.
That reversibility has a compliance consequence. Because the values can be recovered with the key — or re-identified from surrounding context — FPE is pseudonymization, not anonymization. Under GDPR, pseudonymized data is still personal data and stays in scope; FPE lowers exposure, but it does not take the data out of the regulation's reach. Used well, it supports the controls that regimes like PCI DSS and test data call for — keeping cardholder data out of non-production environments while preserving its shape — but it supports those requirements rather than ensuring compliance by itself. In regulated domains where format-sensitive identifiers are everywhere — payment processing, banking, healthcare — that pairing of realistic shape and no real values is what FPE is for. Patterson kept PHI out of its developer workflows and cut test data provisioning time by 75% after automating the de-identification of its production data, so its teams built and debugged against realistic, format-valid data without the sensitive originals ever reaching a lower environment.