JDBC backends sharing a database URL share one pair of compressed-schema tables

[vharseko]

Every JDBC backend maps the compressed-schema trees to the same two tables, whatever its backend id or base DN. Two backends pointed at one database URL therefore share the token space of PersistentCompressedSchema, and the tokens are allocated per instance, so they can hand out the same token for different attribute descriptions and overwrite each other's definitions. Entries already written by the loser then decode with the winner's attribute description.

Surfaced while reviewing #867; it does not belong to that PR.

The tree names carry no backend qualifier

// opendj-server-legacy/src/main/java/org/opends/server/backends/pluggable/PersistentCompressedSchema.java:59
private static final TreeName adTreeName = new TreeName("compressed_schema", DB_NAME_AD);
private static final TreeName ocTreeName = new TreeName("compressed_schema", DB_NAME_OC);

Both are static final with a literal in the baseDN slot. Every other tree of a backend is named from EntryContainer.getTreePrefix(), which is derived from its base DN, so those differ per backend; these two do not.

The JDBC table name is a pure function of that tree name:

// opendj-server-legacy/src/main/java/org/opends/server/backends/jdbc/JDBCStorage.java:157
final LoadingCache<TreeName,String> tree2table = Caffeine.newBuilder()
    .build(treeName -> { ... md.digest(treeName.toString().getBytes()) ... return "opendj_" + hashtext; });

sha224("compressed_schema/compressed_attributes") and sha224("compressed_schema/compressed_object_classes") are constants, so the two table names are constants too — identical for every JDBC backend in every server that shares the database.

That two backends really do share a database is not hypothetical; the test suite already works around it:

// opendj-server-legacy/src/test/java/org/opends/server/backends/jdbc/TestCase.java
/**
 * Backend test classes sharing one database map the same tree names to the same tables,
 * so a previous run may leave trees behind - including entries encrypted with a lost cipher key.
 */
static void dropStaleTrees(Connection con) throws SQLException {

Why sharing them corrupts data

The token is allocated from the size of the instance's own decode map, under the instance's own lock:

// opendj-server-legacy/src/main/java/org/opends/server/api/CompressedSchema.java:296
final Mappings mappings = this.mappings;
id = mappings.adEncodeMap.get(ad);
if (id == null)
{
  id = mappings.adDecodeMap.size();
  mappings.adDecodeMap.add(ad);
  mappings.adEncodeMap.put(ad, id);
  storeAttribute(encodeId(id), ad.getAttributeType().getNameOrOID(), ad.getOptions());
}

There is one PersistentCompressedSchema per RootContainer, i.e. per backend (RootContainer.java:140), and exclusiveLock guards that instance alone. Two backends that loaded the same table at open therefore have the same next token, and the first attribute description each encodes afterwards takes it:

• backend A allocates token N for cn and writes row N → cn;
• backend B allocates token N for sn and overwrites row N → sn.

Both keep serving from their in-memory map, so nothing is visible until a restart. After it, load() (PersistentCompressedSchema.java:139-186) reads N → sn for both, and every entry backend A wrote with token N decodes with the wrong attribute description. The same applies to the object class sets.

Not affected

The Cassandra backend qualifies its table with the backend id, so two Cassandra backends on one keyspace do not collide:

// opendj-server-legacy/src/main/java/org/opends/server/backends/cassandra/CASStorage.java:164
String getTableName() {
    return getKeyspaceName()+".\""+config.getBackendId().replaceAll("[^a-zA-Z0-9_]", "_")+"\"";
}

PDB and JE keep each backend in its own files, so the constant tree name is harmless there — which is presumably why it was never noticed.

Suggested fix

Qualify the two tree names with the same prefix the entry containers use, so they become per-backend, and give the upgrade tooling a rename step — UpgradeTasks.java:860 already renames compressed_schema trees, so the shape exists. getTableName needs no change once the tree name is distinct.

[vharseko]

Reviewed the whole path; every claim above holds. Adding the evidence, one damage path the report does not cover, and four constraints the suggested fix has to satisfy.

Confirmed

The two names are the only new TreeName(...) calls in the tree built from a literal rather than from EntryContainer.getTreePrefix() — which is baseDN.toNormalizedUrlSafeString() (EntryContainer.java:427). The one other literal is the migration task itself (UpgradeTasks.java:860).

The resulting table names, computed:

/compressed_schema/compressed_attributes     -> opendj_e2294f6da66249788dd98ef0eaa7e2e1ac1ecee36e0379978aaba584
/compressed_schema/compressed_object_classes -> opendj_2d5bf8029431e9bfc9219e314541368cb2e08a08ce8271ff032c79b8

They are also unqualified by schema — create table opendj_<hash> (JDBCStorage.java:824) resolves against the connection's default schema.

The corruption mechanics hold too: exclusiveLock is an instance field, not static (CompressedSchema.java:89), so two backends never coordinate; store() issues a bare txn.put (PersistentCompressedSchema.java:207), an upsert with no conditional, so the last writer wins silently; and decodeAttribute raises ERR_COMPRESSEDSCHEMA_UNRECOGNIZED_AD_TOKEN only when the token is absent altogether (CompressedSchema.java:191-197) — a token that resolves to the wrong attribute description decodes without any error at all.

The "not affected" section checks out as well: CASStorage.java:164 puts every tree of a backend into a table named for the backend id, and JE/PDB keep each backend in its own directory.

A second, faster path to the same loss: removeStorageFiles()

The report has two backends racing for the next token, which needs both of them writing and only shows up at the next restart. There is a blunter path.

JDBCStorage.removeStorageFiles() (JDBCStorage.java:653-688) drops every table in listTrees(), and listTrees() is tree2table.asMap().keySet() (JDBCStorage.java:1197) — the process-local Caffeine cache of tree names this process has touched. It is reached from BackendImpl.java:673 whenever importConfig.clearBackend() is set, i.e. import-ldif --clearBackend and ds-task-import with the same flag.

storage is built once in configureBackend (BackendImpl.java:192) and survives closeBackend(); JDBCStorage.close() only flips storageStatus and leaves the cache intact. So if this process ever opened backend A, the two constant tree names are in its cache, and clearing A drops the shared compressed-schema tables out from under backend B while B is running and healthy. B keeps serving from memory and loses the whole mapping at its next start — no interleaved allocation required.

Separately, and outside this issue's scope: the same process-local cache means an offline import-ldif --clearBackend (fresh JDBCStorage, empty cache) drops nothing at all.

Constraints on the fix

1. The qualifier has to be the backend id, not the entry container prefix. PersistentCompressedSchema is constructed once per RootContainer, before any entry container exists (RootContainer.java:140-141), and config.getBaseDN() is a set. A backend holding two base DNs has no single tree prefix to borrow.

2. UpgradeTasks.java:860 is not a reusable shape. That task migrates 2.x local-db backends to pluggable JE: it renames JE databases through a JE Environment directly, and never runs for PDB or JDBC.

3. Changing the constant breaks JE, PDB and Cassandra too, not just JDBC. The tree name is part of the persisted format of all four storages. Flip the literal with no migration and the first start after the upgrade opens an empty tree, restarts token allocation at 0, and mis-decodes every existing entry of every pluggable backend — low tokens silently as the wrong attribute description, high ones as ERR_COMPRESSEDSCHEMA_UNRECOGNIZED_AD_TOKEN. A migration is mandatory, not optional.

4. The read-only open needs its own branch. BackendImpl.java:952 opens the root container READ_ONLY for export-ldif and verify, where load(txn, false) cannot write anything. And in JDBC openTree(name, false) is a no-op with no existence check (JDBCStorage.java:821-822), so a cursor over a table that does not exist fails with a SQL error rather than an empty tree.

Suggested shape

Qualify the tree names with the backend id — as CASStorage already does for its table — and put the migration in PersistentCompressedSchema.load() rather than in the upgrade tooling:

• open /compressed_schema_<backendId>/...; if it is empty and the legacy /compressed_schema/... is not, copy the records across and load from the new one;
• in read-only mode read the legacy tree directly and write nothing;
• do not drop the legacy tree — on a shared JDBC database another backend still needs it, and it is the rollback path.

Written against the SPI, that one migration covers JE, PDB, JDBC and Cassandra at once, and moves two small trees per backend. It does need an existence probe: listTrees() cannot serve as one for JDBC (process-local cache), so either a treeExists(TreeName) on the transaction SPI — JE env.getDatabaseNames(), JDBC the existing private isExistsTable — or catching the failed cursor open.

The alternative, qualifying only JDBCStorage.getTableName, keeps the blast radius inside the storage that has the defect, but renames every table of every existing JDBC backend instead of two, and needs a JDBC-specific migration. The first option looks better on both counts.

On already-damaged installations: copying the shared table into each backend's private tree hands both the same, already-corrupted snapshot. It stops the damage growing; it does not repair entries written before the upgrade. Those need an export/reimport from whichever side is intact.

Workaround available today

Nothing validates getDBDirectory() for uniqueness, so the configuration is accepted silently. Since the tables are unqualified by schema, giving each JDBC backend its own database — or its own schema/user in the URL — keeps them apart until this is fixed.