Methodology — google-transparency-report-analysis.oriz.in

Data Acquisition

All data originates from Google's official Transparency Report at transparencyreport.google.com.
Where Google publishes a bulk CSV/ZIP download, it is fetched directly and cached locally (e.g. copyright removals bulk ZIP at storage.googleapis.com/transparencyreport/).
Where no bulk download exists, figures are read from the interactive report pages. Jina Reader (r.jina.ai/<url>) is used as a free fallback to extract page text where needed.
No browser scraping. No third-party data brokers. No paid APIs.

Processing Pipeline

src/gtra/datasets.py — registry of all datasets with source URLs and bulk availability flags.
src/gtra/download.py — fetches and caches each bulk dataset under data/<id>/. Idempotent — skips if cached.
src/gtra/analyze.py — per-dataset analysis. For copyright: loads CSV, normalizes column names, generates charts. For non-bulk: returns documented stub findings.
src/gtra/charts.py — shared matplotlib/seaborn chart helpers with a consistent editorial palette.
src/gtra/build_site_data.py — exports all findings as JSON to docs/data/ for the static site to consume.

Chart Data

Copyright pages: Chart.js reads from docs/data/copyright.json generated by the Python pipeline. Run python -m gtra.analyze --dataset copyright --download --report then python -m gtra.build_site_data to update.
Non-bulk dataset pages: Charts use manually transcribed values from the interactive report. Values are clearly labelled as approximate and sourced from the official page.
All chart Y-axes show units. All tooltips show exact values.

Limitations

Non-bulk datasets cannot be updated programmatically — they require manual transcription from the interactive report.
Google's bulk CSV for copyright removals does not include the full URL list — only summary counts per request.
Figures for non-bulk datasets are approximate and may lag the latest report by several months.
This project is independent and not endorsed by or affiliated with Google.

# Run full pipeline locally
git clone https://github.com/chirag127/google-transparency-report-analysis.git
cd google-transparency-report-analysis
pip install -r requirements.txt
pip install -e .

# Download + analyze copyright dataset
python -m gtra.analyze --dataset copyright --download --report

# Export all JSON for site
python -m gtra.build_site_data

# Run tests
pip install -r requirements-dev.txt
pytest tests/ --ignore=tests/e2e