Skip to content

docs: correct config precedence and the .secrets/ file format #258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dimitri-yatsenko merged 1 commit into from
Aug 14, 2026
Merged

docs: correct config precedence and the .secrets/ file format #258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions src/how-to/configure-database.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,18 @@ This file should be committed to version control.

## Secrets Directory (`.secrets/`)

Store credentials in `.secrets/datajoint.json`:
Store credentials as one plain-text file per setting:

```json
{
"database.user": "myuser",
"database.password": "mypassword"
}
```bash
mkdir -p .secrets
echo "myuser" > .secrets/database.user
echo "mypassword" > .secrets/database.password
chmod 600 .secrets/*
```

Only `database.user`, `database.password`, and `stores.<name>.<attr>` are read from this
directory — see [Manage Secrets](manage-secrets.md#option-1-secrets-directory-recommended-for-development).

**Important:** Add `.secrets/` to your `.gitignore`:

```gitignore
Expand All @@ -60,7 +63,7 @@ Environment variables take precedence over config files.
## Configuration Settings

| Setting | Environment | Default | Description |
|---------|-------------|---------|-------------|
| --------- | ------------- | --------- | ------------- |
| `database.host` | `DJ_HOST` | `localhost` | Database server hostname |
| `database.port` | `DJ_PORT` | Auto | Database server port (3306 for MySQL, 5432 for PostgreSQL) |
| `database.user` | `DJ_USER` | — | Database username |
Expand Down Expand Up @@ -104,8 +107,8 @@ with dj.config.override(database={'host': 'test-server'}):

1. Programmatic settings (highest priority)
2. Environment variables
3. `.secrets/datajoint.json`
4. `datajoint.json`
3. `datajoint.json`
4. `.secrets/`
5. Default values (lowest priority)

## TLS Configuration
Expand Down Expand Up @@ -304,4 +307,3 @@ staging = dj.Instance(host="staging.example.com", user="dev", password="...")
```

See [Use Isolated Instances](use-instances.md/) for a complete guide.

8 changes: 4 additions & 4 deletions src/how-to/configure-storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ Multiple stores can be configured for different data types or storage tiers. One
DataJoint loads configuration in priority order:

1. **Environment variables** (highest priority)
2. **Secrets directory** (`.secrets/`)
3. **Config file** (`datajoint.json`)
2. **Config file** (`datajoint.json`)
3. **Secrets directory** (`.secrets/`)
4. **Defaults** (lowest priority)

## Single Store Configuration
Expand Down Expand Up @@ -196,7 +196,7 @@ print(dj.config.stores.keys())
## Configuration Options

| Option | Required | Description |
|--------|----------|-------------|
| -------- | ---------- | ------------- |
| `stores.default` | Yes | Name of the default store |
| `stores.<name>.protocol` | Yes | `file`, `s3`, `gcs`, or `azure` |
| `stores.<name>.location` | Yes | Base path or prefix (includes project context) |
Expand Down Expand Up @@ -248,7 +248,7 @@ Schema-addressed storage (`<object@>`, `<npy@>`) does not use subfolding—it us
### Filesystem Recommendations

| Filesystem | Subfolding Needed | Notes |
|------------|-------------------|-------|
| ------------ | ------------------- | ------- |
| ext3 | Yes | Limited directory indexing |
| FAT32/exFAT | Yes | Linear directory scans |
| NFS | Yes | Network latency amplifies directory lookups |
Expand Down
10 changes: 5 additions & 5 deletions src/how-to/deploy-production.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Configure DataJoint for production environments with controlled schema changes a
Development and production environments have different requirements:

| Concern | Development | Production |
|---------|-------------|------------|
| --------- | ------------- | ------------ |
| Schema changes | Automatic table creation | Controlled, explicit changes only |
| Naming | Ad-hoc schema names | Consistent project prefixes |
| Configuration | Local settings | Environment-based |
Expand Down Expand Up @@ -50,7 +50,7 @@ Or in `datajoint.json`:
With `create_tables=False`:

| Action | Development (True) | Production (False) |
|--------|-------------------|-------------------|
| -------- | ------------------- | ------------------- |
| Access existing table | Works | Works |
| Access missing table | Creates it | **Raises error** |
| Explicit `Schema(create_tables=True)` | Creates | Creates (override) |
Expand Down Expand Up @@ -167,8 +167,8 @@ Use different configurations for development, staging, and production.
DataJoint loads settings in priority order:

1. **Environment variables** (highest priority)
2. **Secrets directory** (`.secrets/`)
3. **Config file** (`datajoint.json`)
2. **Config file** (`datajoint.json`)
3. **Secrets directory** (`.secrets/`)
4. **Defaults** (lowest priority)

### Development Setup
Expand Down Expand Up @@ -331,7 +331,7 @@ if __name__ == '__main__':
## Summary

| Setting | Development | Production |
|---------|-------------|------------|
| --------- | ------------- | ------------ |
| `database.create_tables` | `true` | `false` |
| `database.database_prefix` | `""` or `dev_` | `prod_` |
| `safemode` | `true` | `false` (automated) |
Expand Down
64 changes: 32 additions & 32 deletions src/how-to/manage-secrets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Secure configuration management for database credentials, storage access keys, a
DataJoint separates configuration into sensitive and non-sensitive components:

| Component | Location | Purpose | Version Control |
|-----------|----------|---------|-----------------|
| ----------- | ---------- | --------- | ----------------- |
| **Non-sensitive** | `datajoint.json` | Project settings, defaults | ✅ Commit to git |
| **Sensitive** | `.secrets/` directory | Credentials, API keys | ❌ Never commit |
| **Dynamic** | Environment variables | CI/CD, production | ⚠️ Context-dependent |
Expand All @@ -19,7 +19,7 @@ DataJoint loads configuration in this priority order (highest to lowest):
1. **Programmatic settings** — `dj.config['key'] = value`
2. **Environment variables** — `DJ_HOST`, `DJ_USER`, `DJ_STORES`, etc.
3. **Project configuration** — `datajoint.json`
4. **Secrets directory** — `.secrets/stores.<name>.<attr>` (fills attributes the file/env didn't already set)
4. **Secrets directory** — `.secrets/database.user`, `.secrets/database.password`, `.secrets/stores.<name>.<attr>` (each fills a value the file and env didn't already set)
5. **Default values** — Built-in defaults

Higher priority sources override lower ones. Set `DJ_IGNORE_CONFIG_FILE=true` *(new in 2.2.4)* to skip both `datajoint.json` and the secrets directory entirely — see [Env-var-only deployments](#env-var-only-deployments) below.
Expand All @@ -33,7 +33,8 @@ project/
├── datajoint.json # Non-sensitive settings (commit)
├── .gitignore # Must include .secrets/
├── .secrets/
│ ├── datajoint.json # Database credentials
│ ├── database.user # Database credentials, one value per file
│ ├── database.password
│ ├── stores.main.access_key # S3/cloud storage credentials
│ ├── stores.main.secret_key
│ ├── stores.archive.access_key
Expand All @@ -56,13 +57,13 @@ project/

### Option 1: Secrets Directory (Recommended for Development)

Create `.secrets/datajoint.json`:
Each secret is a separate plain-text file named for the setting it carries:

```json
{
"database.user": "myuser",
"database.password": "mypassword"
}
```bash
mkdir -p .secrets
echo "myuser" > .secrets/database.user
echo "mypassword" > .secrets/database.password
chmod 600 .secrets/*
```

Non-sensitive database settings go in `datajoint.json`:
Expand Down Expand Up @@ -122,7 +123,7 @@ Local or network-mounted file systems don't require credentials:

### S3/MinIO Storage (With Credentials)

#### Config in `datajoint.json` (non-sensitive):
#### Config in `datajoint.json` (non-sensitive)

```json
{
Expand All @@ -144,7 +145,7 @@ Local or network-mounted file systems don't require credentials:
}
```

#### Credentials in `.secrets/` directory:
#### Credentials in `.secrets/` directory

Create separate files for each store's credentials:

Expand Down Expand Up @@ -202,7 +203,7 @@ If `DJ_STORES` contains invalid JSON, DataJoint raises `ValueError` at config-lo
### Database Connections

| Setting | Environment Variable | Description |
|---------|---------------------|-------------|
| --------- | --------------------- | ------------- |
| `database.host` | `DJ_HOST` | Database hostname |
| `database.port` | `DJ_PORT` | Database port (default: 3306) |
| `database.user` | `DJ_USER` | Database username |
Expand Down Expand Up @@ -256,16 +257,12 @@ chmod 700 .secrets # Owner-only access
# 2. Create .gitignore
echo ".secrets/" >> .gitignore

# 3. Store credentials in .secrets/
cat > .secrets/datajoint.json <<EOF
{
"database.user": "dev_user",
"database.password": "dev_password"
}
EOF
# 3. Store credentials in .secrets/, one value per file
echo "dev_user" > .secrets/database.user
echo "dev_password" > .secrets/database.password

# 4. Set restrictive permissions
chmod 600 .secrets/datajoint.json
chmod 600 .secrets/*
```

### Production Environment
Expand Down Expand Up @@ -370,7 +367,7 @@ import datajoint as dj

# Config loaded automatically from:
# 1. datajoint.json (project settings)
# 2. .secrets/datajoint.json (credentials)
# 2. .secrets/database.user and .secrets/database.password (credentials)
conn = dj.conn()
```

Expand Down Expand Up @@ -401,6 +398,10 @@ project/

**Load by environment:**

These per-environment JSON files are read by the snippet below, not by DataJoint's own
secrets loader, so they can hold any settings and use any filenames. Assigning through
`dj.config[...]` is a programmatic override, which outranks every other source.

```python
import os
import datajoint as dj
Expand Down Expand Up @@ -482,7 +483,7 @@ conn = dj.conn(reset=True)
git history; until rotation completes, the leaked secret is still valid.

- Database users (`database.user` / `database.password`): change the password
on the server, then update your local `.secrets/datajoint.json`.
on the server, then update your local `.secrets/database.password`.
- Object-store credentials (`stores.<name>.access_key` / `secret_key`, or the
equivalent in your cloud provider): issue new keys and revoke the old ones.
- Any third-party tokens that appeared in the same file.
Expand All @@ -493,9 +494,9 @@ git history; until rotation completes, the leaked secret is still valid.
clones don't leak it further:

```bash
# Remove file from history
# Remove the leaked file from history (repeat per file, or pass several paths)
git filter-branch --force --index-filter \
"git rm --cached --ignore-unmatch .secrets/datajoint.json" \
"git rm --cached --ignore-unmatch .secrets/database.password" \
--prune-empty --tag-name-filter cat -- --all

# Force push (coordinate with team!)
Expand All @@ -505,7 +506,7 @@ git push origin --force --all
**Step 3 — Verify removal:**

```bash
git log --all --full-history -- .secrets/datajoint.json
git log --all --full-history -- .secrets/database.password
```

## Configuration Templates
Expand All @@ -520,12 +521,10 @@ git log --all --full-history -- .secrets/datajoint.json
}
```

```json
// .secrets/datajoint.json
{
"database.user": "root",
"database.password": "simple"
}
```
// .secrets/ directory
.secrets/database.user # root
.secrets/database.password # simple
```

### Production with S3 Storage
Expand All @@ -551,7 +550,8 @@ git log --all --full-history -- .secrets/datajoint.json

```
// .secrets/ directory
.secrets/datajoint.json # Database credentials
.secrets/database.user # Database username
.secrets/database.password # Database password
.secrets/stores.main.access_key # S3 access key
.secrets/stores.main.secret_key # S3 secret key
```
Expand Down
Loading
Loading