From 723a33252c0a947236c525850dd6ffab1cb00c0e Mon Sep 17 00:00:00 2001 From: George Touloumes Date: Thu, 13 Aug 2026 15:32:56 -0400 Subject: [PATCH] docs: correct config precedence and the .secrets/ file format MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects, both verified against datajoint-python 2.3 source and by running the loader against fixtures. **Precedence was inverted in four pages.** `datajoint.json` outranks `.secrets/`, not the other way round. Two mechanisms in settings.py enforce it: `_update_from_flat_dict` skips a file value when its env var is set ("env var takes precedence"), and `_load_secrets` assigns only when the target is still unset, with the comment "Only set if not already present (config / env vars win)". Confirmed empirically — with the same key in all three sources, `DJ_USER` wins, then datajoint.json, and `.secrets/` supplies a value only when nothing else did. The actual order is: programmatic, environment variables, datajoint.json, .secrets/, defaults. Fixed in reference/configuration.md, how-to/configure-storage.md, how-to/deploy-production.md, and how-to/configure-database.md. manage-secrets.md and specs/object-store-configuration.md already had it right. Note the module docstring in datajoint-python's settings.py states the inverted order and contradicts its own implementation twenty lines below; that is the likely source of the error here and needs an upstream fix. **`.secrets/datajoint.json` is never read.** The secrets directory is read file by file: only `database.user`, `database.password`, and `stores..` are recognized. A JSON file there contributes nothing, so the documented "recommended for development" setup left credentials unset and produced the same access-denied error the format was meant to avoid. Replaced with per-key files in manage-secrets.md (Option 1, the directory tree, the best-practices block, both templates, and the leaked-credential rotation and history-scrubbing steps), configure-database.md, and migrate-to-v20.md — which additionally told migrators to put database.host in the secrets directory, a key that directory does not read; the host moves to datajoint.json. The Multi-Environment pattern keeps its JSON files: that snippet loads them itself and assigns through dj.config, which works regardless. --- src/how-to/configure-database.md | 22 ++++++----- src/how-to/configure-storage.md | 8 ++-- src/how-to/deploy-production.md | 10 ++--- src/how-to/manage-secrets.md | 64 ++++++++++++++++---------------- src/how-to/migrate-to-v20.md | 36 +++++++++--------- src/reference/configuration.md | 24 ++++++------ 6 files changed, 83 insertions(+), 81 deletions(-) diff --git a/src/how-to/configure-database.md b/src/how-to/configure-database.md index 64183ea8..1cd3298a 100644 --- a/src/how-to/configure-database.md +++ b/src/how-to/configure-database.md @@ -30,15 +30,18 @@ This file should be committed to version control. ## Secrets Directory (`.secrets/`) -Store credentials in `.secrets/datajoint.json`: +Store credentials as one plain-text file per setting: -```json -{ - "database.user": "myuser", - "database.password": "mypassword" -} +```bash +mkdir -p .secrets +echo "myuser" > .secrets/database.user +echo "mypassword" > .secrets/database.password +chmod 600 .secrets/* ``` +Only `database.user`, `database.password`, and `stores..` are read from this +directory — see [Manage Secrets](manage-secrets.md#option-1-secrets-directory-recommended-for-development). + **Important:** Add `.secrets/` to your `.gitignore`: ```gitignore @@ -60,7 +63,7 @@ Environment variables take precedence over config files. ## Configuration Settings | Setting | Environment | Default | Description | -|---------|-------------|---------|-------------| +| --------- | ------------- | --------- | ------------- | | `database.host` | `DJ_HOST` | `localhost` | Database server hostname | | `database.port` | `DJ_PORT` | Auto | Database server port (3306 for MySQL, 5432 for PostgreSQL) | | `database.user` | `DJ_USER` | — | Database username | @@ -104,8 +107,8 @@ with dj.config.override(database={'host': 'test-server'}): 1. Programmatic settings (highest priority) 2. Environment variables -3. `.secrets/datajoint.json` -4. `datajoint.json` +3. `datajoint.json` +4. `.secrets/` 5. Default values (lowest priority) ## TLS Configuration @@ -304,4 +307,3 @@ staging = dj.Instance(host="staging.example.com", user="dev", password="...") ``` See [Use Isolated Instances](use-instances.md/) for a complete guide. - diff --git a/src/how-to/configure-storage.md b/src/how-to/configure-storage.md index 55f4ea01..06579bff 100644 --- a/src/how-to/configure-storage.md +++ b/src/how-to/configure-storage.md @@ -26,8 +26,8 @@ Multiple stores can be configured for different data types or storage tiers. One DataJoint loads configuration in priority order: 1. **Environment variables** (highest priority) -2. **Secrets directory** (`.secrets/`) -3. **Config file** (`datajoint.json`) +2. **Config file** (`datajoint.json`) +3. **Secrets directory** (`.secrets/`) 4. **Defaults** (lowest priority) ## Single Store Configuration @@ -196,7 +196,7 @@ print(dj.config.stores.keys()) ## Configuration Options | Option | Required | Description | -|--------|----------|-------------| +| -------- | ---------- | ------------- | | `stores.default` | Yes | Name of the default store | | `stores..protocol` | Yes | `file`, `s3`, `gcs`, or `azure` | | `stores..location` | Yes | Base path or prefix (includes project context) | @@ -248,7 +248,7 @@ Schema-addressed storage (``, ``) does not use subfolding—it us ### Filesystem Recommendations | Filesystem | Subfolding Needed | Notes | -|------------|-------------------|-------| +| ------------ | ------------------- | ------- | | ext3 | Yes | Limited directory indexing | | FAT32/exFAT | Yes | Linear directory scans | | NFS | Yes | Network latency amplifies directory lookups | diff --git a/src/how-to/deploy-production.md b/src/how-to/deploy-production.md index 9612c234..d8ef13b8 100644 --- a/src/how-to/deploy-production.md +++ b/src/how-to/deploy-production.md @@ -7,7 +7,7 @@ Configure DataJoint for production environments with controlled schema changes a Development and production environments have different requirements: | Concern | Development | Production | -|---------|-------------|------------| +| --------- | ------------- | ------------ | | Schema changes | Automatic table creation | Controlled, explicit changes only | | Naming | Ad-hoc schema names | Consistent project prefixes | | Configuration | Local settings | Environment-based | @@ -50,7 +50,7 @@ Or in `datajoint.json`: With `create_tables=False`: | Action | Development (True) | Production (False) | -|--------|-------------------|-------------------| +| -------- | ------------------- | ------------------- | | Access existing table | Works | Works | | Access missing table | Creates it | **Raises error** | | Explicit `Schema(create_tables=True)` | Creates | Creates (override) | @@ -167,8 +167,8 @@ Use different configurations for development, staging, and production. DataJoint loads settings in priority order: 1. **Environment variables** (highest priority) -2. **Secrets directory** (`.secrets/`) -3. **Config file** (`datajoint.json`) +2. **Config file** (`datajoint.json`) +3. **Secrets directory** (`.secrets/`) 4. **Defaults** (lowest priority) ### Development Setup @@ -331,7 +331,7 @@ if __name__ == '__main__': ## Summary | Setting | Development | Production | -|---------|-------------|------------| +| --------- | ------------- | ------------ | | `database.create_tables` | `true` | `false` | | `database.database_prefix` | `""` or `dev_` | `prod_` | | `safemode` | `true` | `false` (automated) | diff --git a/src/how-to/manage-secrets.md b/src/how-to/manage-secrets.md index e423b1c5..07fe2d22 100644 --- a/src/how-to/manage-secrets.md +++ b/src/how-to/manage-secrets.md @@ -7,7 +7,7 @@ Secure configuration management for database credentials, storage access keys, a DataJoint separates configuration into sensitive and non-sensitive components: | Component | Location | Purpose | Version Control | -|-----------|----------|---------|-----------------| +| ----------- | ---------- | --------- | ----------------- | | **Non-sensitive** | `datajoint.json` | Project settings, defaults | ✅ Commit to git | | **Sensitive** | `.secrets/` directory | Credentials, API keys | ❌ Never commit | | **Dynamic** | Environment variables | CI/CD, production | ⚠️ Context-dependent | @@ -19,7 +19,7 @@ DataJoint loads configuration in this priority order (highest to lowest): 1. **Programmatic settings** — `dj.config['key'] = value` 2. **Environment variables** — `DJ_HOST`, `DJ_USER`, `DJ_STORES`, etc. 3. **Project configuration** — `datajoint.json` -4. **Secrets directory** — `.secrets/stores..` (fills attributes the file/env didn't already set) +4. **Secrets directory** — `.secrets/database.user`, `.secrets/database.password`, `.secrets/stores..` (each fills a value the file and env didn't already set) 5. **Default values** — Built-in defaults Higher priority sources override lower ones. Set `DJ_IGNORE_CONFIG_FILE=true` *(new in 2.2.4)* to skip both `datajoint.json` and the secrets directory entirely — see [Env-var-only deployments](#env-var-only-deployments) below. @@ -33,7 +33,8 @@ project/ ├── datajoint.json # Non-sensitive settings (commit) ├── .gitignore # Must include .secrets/ ├── .secrets/ -│ ├── datajoint.json # Database credentials +│ ├── database.user # Database credentials, one value per file +│ ├── database.password │ ├── stores.main.access_key # S3/cloud storage credentials │ ├── stores.main.secret_key │ ├── stores.archive.access_key @@ -56,13 +57,13 @@ project/ ### Option 1: Secrets Directory (Recommended for Development) -Create `.secrets/datajoint.json`: +Each secret is a separate plain-text file named for the setting it carries: -```json -{ - "database.user": "myuser", - "database.password": "mypassword" -} +```bash +mkdir -p .secrets +echo "myuser" > .secrets/database.user +echo "mypassword" > .secrets/database.password +chmod 600 .secrets/* ``` Non-sensitive database settings go in `datajoint.json`: @@ -122,7 +123,7 @@ Local or network-mounted file systems don't require credentials: ### S3/MinIO Storage (With Credentials) -#### Config in `datajoint.json` (non-sensitive): +#### Config in `datajoint.json` (non-sensitive) ```json { @@ -144,7 +145,7 @@ Local or network-mounted file systems don't require credentials: } ``` -#### Credentials in `.secrets/` directory: +#### Credentials in `.secrets/` directory Create separate files for each store's credentials: @@ -202,7 +203,7 @@ If `DJ_STORES` contains invalid JSON, DataJoint raises `ValueError` at config-lo ### Database Connections | Setting | Environment Variable | Description | -|---------|---------------------|-------------| +| --------- | --------------------- | ------------- | | `database.host` | `DJ_HOST` | Database hostname | | `database.port` | `DJ_PORT` | Database port (default: 3306) | | `database.user` | `DJ_USER` | Database username | @@ -256,16 +257,12 @@ chmod 700 .secrets # Owner-only access # 2. Create .gitignore echo ".secrets/" >> .gitignore -# 3. Store credentials in .secrets/ -cat > .secrets/datajoint.json < .secrets/database.user +echo "dev_password" > .secrets/database.password # 4. Set restrictive permissions -chmod 600 .secrets/datajoint.json +chmod 600 .secrets/* ``` ### Production Environment @@ -370,7 +367,7 @@ import datajoint as dj # Config loaded automatically from: # 1. datajoint.json (project settings) -# 2. .secrets/datajoint.json (credentials) +# 2. .secrets/database.user and .secrets/database.password (credentials) conn = dj.conn() ``` @@ -401,6 +398,10 @@ project/ **Load by environment:** +These per-environment JSON files are read by the snippet below, not by DataJoint's own +secrets loader, so they can hold any settings and use any filenames. Assigning through +`dj.config[...]` is a programmatic override, which outranks every other source. + ```python import os import datajoint as dj @@ -482,7 +483,7 @@ conn = dj.conn(reset=True) git history; until rotation completes, the leaked secret is still valid. - Database users (`database.user` / `database.password`): change the password - on the server, then update your local `.secrets/datajoint.json`. + on the server, then update your local `.secrets/database.password`. - Object-store credentials (`stores..access_key` / `secret_key`, or the equivalent in your cloud provider): issue new keys and revoke the old ones. - Any third-party tokens that appeared in the same file. @@ -493,9 +494,9 @@ git history; until rotation completes, the leaked secret is still valid. clones don't leak it further: ```bash -# Remove file from history +# Remove the leaked file from history (repeat per file, or pass several paths) git filter-branch --force --index-filter \ - "git rm --cached --ignore-unmatch .secrets/datajoint.json" \ + "git rm --cached --ignore-unmatch .secrets/database.password" \ --prune-empty --tag-name-filter cat -- --all # Force push (coordinate with team!) @@ -505,7 +506,7 @@ git push origin --force --all **Step 3 — Verify removal:** ```bash -git log --all --full-history -- .secrets/datajoint.json +git log --all --full-history -- .secrets/database.password ``` ## Configuration Templates @@ -520,12 +521,10 @@ git log --all --full-history -- .secrets/datajoint.json } ``` -```json -// .secrets/datajoint.json -{ - "database.user": "root", - "database.password": "simple" -} +``` +// .secrets/ directory +.secrets/database.user # root +.secrets/database.password # simple ``` ### Production with S3 Storage @@ -551,7 +550,8 @@ git log --all --full-history -- .secrets/datajoint.json ``` // .secrets/ directory -.secrets/datajoint.json # Database credentials +.secrets/database.user # Database username +.secrets/database.password # Database password .secrets/stores.main.access_key # S3 access key .secrets/stores.main.secret_key # S3 secret key ``` diff --git a/src/how-to/migrate-to-v20.md b/src/how-to/migrate-to-v20.md index 4b67057d..07839ccc 100644 --- a/src/how-to/migrate-to-v20.md +++ b/src/how-to/migrate-to-v20.md @@ -14,7 +14,7 @@ Upgrade existing pipelines from legacy DataJoint (pre-2.0) to DataJoint 2.0+. ### System Requirements | Component | Legacy (pre-2.0) | DataJoint 2.0+ | -|-----------|-----------------|---------------| +| ----------- | ----------------- | --------------- | | **Python** | 3.8+ | **3.10+** | | **MySQL** | 5.7+ | **8.0+** | | **Character encoding** | (varies) | **UTF-8 (utf8mb4)** | @@ -83,7 +83,7 @@ Tests provide immediate ROI during migration and ongoing value for development. DataJoint 2.0 introduces a unified type system with three tiers: | Tier | Description | Examples | Migration | -|------|-------------|----------|-----------| +| ------ | ------------- | ---------- | ----------- | | **Native** | Raw MySQL types | `int unsigned`, `tinyint` | Auto-converted to core types | | **Core** | Standardized portable types | `int64`, `float64`, `varchar(100)`, `json` | Phase I | | **Codec** | Serialization to blob or storage | ``, ``, `` | Phase I-III | @@ -97,7 +97,7 @@ DataJoint 2.0 makes serialization **explicit** with codecs. In pre-2.0, `longblo #### Migration: Legacy → 2.0 | pre-2.0 (Implicit) | 2.0 (Explicit) | Storage | Migration | -|-------------------|----------------|---------|-----------| +| ------------------- | ---------------- | --------- | ----------- | | `longblob` | `` | In-table | Phase I code, Phase III data | | `mediumblob` | `` | In-table | Phase I code, Phase III data | | `blob`, `tinyblob` | `` | In-table | Phase I code, Phase III data | @@ -142,7 +142,7 @@ COMMENT '::large array in object storage' In pre-2.0, `longblob` columns automatically deserialized Python objects using DataJoint's binary serialization format. DataJoint 2.0 identifies blob columns by checking for `::` in the column comment. **Without this marker, blob columns are treated as raw binary data and will NOT be deserialized.** | Column Comment | DataJoint 2.0 Behavior | -|----------------|----------------------| +| ---------------- | ---------------------- | | `::neural data` | ✓ Deserializes to Python/NumPy objects | | `neural data` (no marker) | ✗ Returns raw bytes (no deserialization) | @@ -237,7 +237,7 @@ DataJoint 2.0 replaces `external.*` with unified `stores.*` configuration: ### Query API Changes | pre-2.0 | 2.0 | Phase | -|--------|-----|-------| +| -------- | ----- | ------- | | `table.fetch()` | `table.to_arrays()` or `table.to_dicts()` | I | | `table.fetch(..., format="frame")` | `table.to_pandas(...)` | I | | `table.fetch1()` | `table.fetch1()` (unchanged) | — | @@ -259,7 +259,7 @@ DataJoint 2.0 replaces `external.*` with unified `stores.*` configuration: ## Migration Overview | Phase | Goal | Code Changes | Schema/Store Changes | Production Impact | -|-------|------|--------------|----------------------|-------------------| +| ------- | ------ | -------------- | ---------------------- | ------------------- | | **I** | Branch & code migration | All API updates, type syntax, **all codecs** (in-table and in-store) | Empty `_v2` schemas + test stores | **None** | | **II** | Test compatibility | — | Populate `_v2` schemas with sample data, test equivalence | **None** | | **III** | Migrate production data | — | Multiple migration options | **Varies** | @@ -690,7 +690,7 @@ Now configure database connection and stores. DataJoint 2.0 uses: -- **`.secrets/datajoint.json`** for credentials (gitignored) +- **`.secrets/database.user`** and **`.secrets/database.password`** for credentials (gitignored) - **`datajoint.json`** for non-sensitive settings (checked in) - **`stores.*`** instead of `external.*` @@ -707,18 +707,18 @@ echo ".secrets/" >> .gitignore python -c "import datajoint as dj; dj.config.save_template()" ``` -**Edit `.secrets/datajoint.json`:** -```json -{ - "database.host": "your-database-host", - "database.user": "your-username", - "database.password": "your-password" -} +**Fill in the credential files under `.secrets/`** — one value per file: +```bash +echo "your-username" > .secrets/database.user +echo "your-password" > .secrets/database.password ``` -**Edit `datajoint.json`:** +**Edit `datajoint.json`**: ```json { + "database": { + "host": "your-database-host" + }, "loglevel": "INFO", "safemode": true, "display.limit": 12, @@ -1077,7 +1077,7 @@ Convert ALL types and codecs in Phase I: **Integer and Float Types:** | pre-2.0 | 2.0 | Category | -|--------|-----|----------| +| -------- | ----- | ---------- | | `int unsigned` | `int64` | Core type | | `int` | `int32` | Core type | | `smallint unsigned` | `int32` | Core type | @@ -1113,7 +1113,7 @@ precision, so converting to `decimal(M,D)` is the recommended move. **String, Date, and Structured Types:** | pre-2.0 | 2.0 | Notes | -|--------|-----|-------| +| -------- | ----- | ------- | | `varchar(N)`, `char(N)` | Unchanged | Core types | | `date` | Unchanged | Core type | | `enum('a', 'b')` | Unchanged | Core type | @@ -1129,7 +1129,7 @@ precision, so converting to `decimal(M,D)` is the recommended move. **Codecs:** | pre-2.0 | 2.0 | Category | -|--------|-----|----------| +| -------- | ----- | ---------- | | `longblob` | `` | Codec (in-table) | | `attach` | `` | Codec (in-table) | | `blob@store` | `` | Codec (in-store) | diff --git a/src/reference/configuration.md b/src/reference/configuration.md index a1a08f03..73a13f52 100644 --- a/src/reference/configuration.md +++ b/src/reference/configuration.md @@ -9,14 +9,14 @@ DataJoint configuration options and settings. Configuration is loaded in priority order: 1. **Environment variables** (highest priority) -2. **Secrets directory** (`.secrets/`) -3. **Config file** (`datajoint.json`) +2. **Config file** (`datajoint.json`) +3. **Secrets directory** (`.secrets/`) 4. **Defaults** (lowest priority) ## Database Settings | Setting | Environment | Default | Description | -|---------|-------------|---------|-------------| +| --------- | ------------- | --------- | ------------- | | `database.backend` | `DJ_BACKEND` | `mysql` | Database backend: `mysql` or `postgresql` *(new in 2.1)* | | `database.host` | `DJ_HOST` | `localhost` | Database server hostname | | `database.port` | `DJ_PORT` | `3306`/`5432` | Database server port (auto-detects from backend) | @@ -53,7 +53,7 @@ DataJoint uses two default settings to reflect the architectural distinction bet **Common settings (all protocols):** | Setting | Required | Description | -|---------|----------|-------------| +| --------- | ---------- | ------------- | | `stores..protocol` | Yes | Storage protocol: `file`, `s3`, `gcs`, `azure` | | `stores..location` | Yes | Base path or prefix (includes project context) | | `stores..hash_prefix` | No | Path prefix for hash-addressed section (default: `"_hash"`) | @@ -90,7 +90,7 @@ Prefixes must be mutually exclusive (no prefix can be a parent/child of another) **S3-specific settings:** | Setting | Required | Description | -|---------|----------|-------------| +| --------- | ---------- | ------------- | | `stores..endpoint` | Yes | S3 endpoint URL (e.g., `s3.amazonaws.com`) | | `stores..bucket` | Yes | Bucket name | | `stores..access_key` | Yes | S3 access key ID | @@ -100,7 +100,7 @@ Prefixes must be mutually exclusive (no prefix can be a parent/child of another) **GCS-specific settings:** | Setting | Required | Description | -|---------|----------|-------------| +| --------- | ---------- | ------------- | | `stores..bucket` | Yes | GCS bucket name | | `stores..token` | Yes | Authentication token path | | `stores..project` | No | GCS project ID | @@ -108,7 +108,7 @@ Prefixes must be mutually exclusive (no prefix can be a parent/child of another) **Azure-specific settings:** | Setting | Required | Description | -|---------|----------|-------------| +| --------- | ---------- | ------------- | | `stores..container` | Yes | Azure container name | | `stores..account_name` | Yes | Storage account name | | `stores..account_key` | Yes | Storage account key | @@ -154,7 +154,7 @@ If table lacks partition attributes, it follows normal path structure. ## Jobs Settings | Setting | Default | Description | -|---------|---------|-------------| +| --------- | --------- | ------------- | | `jobs.auto_refresh` | `True` | Auto-refresh job queue on populate | | `jobs.keep_completed` | `False` | Retain success records in jobs table | | `jobs.stale_timeout` | `3600` | Seconds before stale job cleanup | @@ -166,7 +166,7 @@ If table lacks partition attributes, it follows normal path structure. ## Display Settings | Setting | Environment | Default | Description | -|---------|-------------|---------|-------------| +| --------- | ------------- | --------- | ------------- | | `display.limit` | — | `12` | Max rows to display | | `display.width` | — | `14` | Column width | | `display.show_tuple_count` | — | `True` | Show row count in output | @@ -175,7 +175,7 @@ If table lacks partition attributes, it follows normal path structure. ## Top-Level Settings | Setting | Environment | Default | Description | -|---------|-------------|---------|-------------| +| --------- | ------------- | --------- | ------------- | | `loglevel` | `DJ_LOG_LEVEL` | `INFO` | Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` | | `safemode` | — | `True` | Require confirmation for destructive operations | | `enable_python_native_blobs` | — | `True` | Allow Python-native blob serialization | @@ -326,7 +326,7 @@ schema = inst.Schema("my_schema") ### Parameters | Parameter | Type | Default | Description | -|-----------|------|---------|-------------| +| ----------- | ------ | --------- | ------------- | | `host` | str | — | Database hostname (required) | | `user` | str | — | Database username (required) | | `password` | str | — | Database password (required) | @@ -337,7 +337,7 @@ schema = inst.Schema("my_schema") ### Attributes and Methods | Member | Description | -|--------|-------------| +| -------- | ------------- | | `inst.config` | This Instance's Config object | | `inst.connection` | This Instance's Connection object | | `inst.Schema(name)` | Create a Schema bound to this Instance |