Hyper-personalised short-video feeds have become one of the main ways people encounter culture, news, and each other. The recommender systems behind them decide what large audiences see each day, and their influence now reaches deep into society, culture, and commerce.
That influence is hard to study from the outside. What is actually in a given person's feed? How does it change over the weeks they spend with it? Do two people who share an interest end up seeing much the same thing, or something quite different?
The For You Data Hub helps researchers answer questions like these by examining feeds just as the people using them are experiencing them — on TikTok, Instagram Reels, and YouTube Shorts. The data used in the Hub are donated by real platform users. Participants have requested their own data export from the platform, reviewed it, and decided what they want to share with the research team. Read more about ethical considerations in docs/ethics_and_data_handling.md.
From there, the Hub carries a donation through the whole pipeline. It reads datasets from different platforms into a single activity table, enriches every watched item with its metadata and media, annotates content with multimodal AI models, and opens the result to analysis — both cross-sectional, comparing participants and groups, and temporal, following how one person's feed shifts day by day or video by video. Read more about the pipeline in docs/pipeline.md.
The For You Data Hub's User Guide is available here docs/user-guide.md, where you can read about the different analyses the Hub allows.
The For You Data Hub is built for academic researchers with high expectations for transparency. Ingestion produces a per-file intake report (rows read, rows kept, plain-language drop reasons), and every study carries an auto-generated methods/provenance note — filters, sample sizes, and the exact annotation/contract versions behind the data — surfaced in the dashboard and exportable as JSON. AI annotation is driven by contracts to maximise replicability and transparency. Researchers can experiment with different models and prompts and run evaluations with human input to ensure coding reliability and validity.
The Hub currently supports the three largest short-video platforms, but it is essentially platform-agnostic. Researchers can extend the Hub to support new platforms by adding a new ingestion class, scraper class, and a contract block (the complete checklist is in docs/extending.md).
| Path | What it is |
|---|---|
fyp/ |
Core Python package: ingestion, scraping, annotation, recoding, analysis |
web_interface/ |
Flask app (dashboard + API) and background worker scripts (run_*.py) |
config/ |
config.toml plus four declarative TOML contracts that own the variable schemas |
tests/ |
unit/ (pytest suite) and golden/ (cost-free annotation regression suite) — both run by CI |
scripts/ |
Setup, verification (verify.sh), demo-data generation, and doc generators |
docs/ |
Documentation: architecture, configuration, web layer, pipeline |
DEVELOPING.md is the maintainer guide: environment, coding style, module
layout, key patterns, and deployment. It is the most detailed single
reference in the repository and the best starting point for contributors.
You can take the Hub for a spin straight away by requesting a user account at https://www.tinyurl.com/foryoudatahub. If you prefer to have you own installation, you can run the server both on your local computer and on Google Cloud. This quickstart explains how to make a local installation.
Make sure you have Python 3.12 (matches the production runtime), plus ffmpeg and
node or deno if you run the scrapers.
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt # runtime pins + pytest/ruff/pre-commit
pip install -e . # recommended: editable install of the fyp package
python scripts/setup.py # interactive setup wizard → config/config.local.toml
python web_interface/fyp_data_hub.py # → http://localhost:5002Your first boot after installation prints a one-time random password for the default
admin@admin.net account — copy it from the console and change it after
logging in. Data storage defaults to ~/fyp_local on the local disk; the
wizard can point it elsewhere or enable GCS/Gemini. (Manual alternative to
the wizard: cp config/config.local.toml.example config/config.local.toml
and edit it.) The full walkthrough — prerequisites per platform, optional
services, first data upload — is in
docs/installation.md. Environment variables (Gemini
key, GCS bucket, ...) are documented in .env.example; a .env file at the
project root is loaded automatically at startup. The wizard can also install
the dependencies for you (--install), and python scripts/setup.py --verify live-checks the configured services afterwards. Installed without
Gemini and want annotation later? See
Enabling Gemini later — no
reinstall needed and your data is untouched.
The editable install is recommended but never required — the app also runs
from a plain checkout (cwd imports and the workers' sys.path bootstrap keep
working, and the Docker image installs nothing from pyproject.toml). Note
that reusing fyp in another project requires a config file: either a
project root containing __proj__.py and config/config.toml, or the
FYP_CONFIG_PATH environment variable pointing at a config TOML directly.
Configuration loads lazily, on first use rather than at import.
Background workers run as plain subprocesses locally (started from the web UI's Data Pipeline tab, or manually):
python web_interface/run_queue_annotator.py
python web_interface/run_queue_scraper.py --platform tiktokEvery change should pass the gate before merging:
source .venv/bin/activate
bash scripts/verify.shIt runs ruff, the checkout-only unit-test subset, the var-schema hash guard, the golden annotation safety net (replays saved Gemini responses — no API cost), and an app import smoke test. See CONTRIBUTING.md for the details and the test markers.
Production runs on Google Cloud Run as two services sharing one Docker
image: fyp-data-hub (web) and fyp-task-runner (background Cloud Tasks).
Storage is Google Cloud Storage; locally it is the filesystem — both behind
the same fyp/core/data_io.py abstraction. Build/deploy commands and the
base-image/app-image split are documented in DEVELOPING.md §"Running the
Project" and docs/architecture.md.
- docs/installation.md — installing from scratch: prerequisites, setup wizard, first run
- docs/architecture.md — system overview, key design patterns
- docs/configuration.md — config.toml sections, contracts, environment variables
- docs/pipeline.md — ingestion → scrape → annotation → recode → analysis
- docs/user-guide.md — the web app, tab by tab, for researchers and students
- docs/contracts.md — the contract system: authoring, validation, versioning, runtime editing
- docs/extending.md — adding a platform, an annotation backend, or an embedding backend
- docs/web_interface.md — Flask app structure, auth, workers, route inventory
- docs/routes.md — generated HTTP endpoint inventory
- docs/correlations-tab-guide.md — the Correlations tab: statistics, views, interpretation
- docs/annotation-ab-findings.md — A/B evidence behind the shipped annotation generation settings (historical)
- docs/ethics_and_data_handling.md — consent, data handling, and the ethics posture of the software
- DEVELOPING.md — the maintainer guide: environment, module layout, key patterns, deployment
- CONTRIBUTING.md — workflow, coding style, invariants you must not break
- SECURITY.md — reporting vulnerabilities
- CHANGELOG.md — release history
MIT — see LICENSE. If you use The For You Data Hub in your research, please cite it using the metadata in CITATION.cff.
The For You Data Hub was developed with substantial assistance from Anthropic's Claude models. Claude Code was used for code generation based on Wikstrom's designs as well as for refactoring, test scaffolding, and drafting of documentation. Wikstrom framed the research problems, designed the architecture and its central abstractions — the versioned contract system, the backend interfaces, the validation harnesses — and reviewed, tested and accepted every change. Responsibility for the correctness, originality and licensing of the code rests with Patrik Wikstrom.
Distinct from that, large language models are also runtime components of the Hub: content annotation and text embedding are performed by Gemini or by open-weight Qwen and MiniCPM models, depending on configuration. That is a function of the software rather than an authoring aid, and the A/B evaluation and human-coding harnesses described in docs/pipeline.md exist precisely to make those model outputs auditable rather than taken on trust.