SHAB Search - Demonstrator

Admin guide

How to run, update, and operate this SHAB search demonstrator. For searching the index, use the user guide.

Sources Stack Start the GUI Daily update Full rebuild Company metadata Operations

SHAB, Zefix, and LINDAS

SHAB (Schweizerisches Handelsamtsblatt) is the Swiss Official Gazette of Commerce. Notices are ingested from ItemStorage (SHAB gatherer).

Zefix (Zentraler Firmenindex) is the Confederation’s commercial-register search. Optional SHAB_DAILY_ZEFIX=1 calls Zefix REST for companies LINDAS does not resolve. GUI register links point at Zefix when seat and EHRAID are known.

LINDAS (Linked Data Service of the Swiss Confederation) is the open SPARQL catalogue we query for CHE↔CH-ID aliases, purpose, legal form, seat, EHRAID, street address, and other registered names (de/fr/it/en).

Stack

The GUI and REST API read Postgres table shab_entities. Source items come from ItemStorage (SHAB gatherer). Parsed JSONL lives under data/shab/. Company UID/CH-ID aliases and LINDAS enrichment live under data/shab/aliases/.

Settings: config/app.ini (host, port, table). Credentials: config/secrets.ini, merged over sibling OpenSanctions secrets when present. Search uses [database_opensanctions]; fetch uses [database_itemstorage] or [database_kyc].

Start the GUI

cd /data/projects/TOYS/toys/trunk/shab-search
source venv/bin/activate
python run.py

Default URL is the host and port in app.ini (this instance uses port 5088). Keep debug = false on this host. After Python changes, restart the process. After CSS or JS changes, hard-refresh the browser. Health: /api/health.

Daily update

Incremental ingest does not re-parse the full dump. Cron example:

30 3 * * * /data/projects/TOYS/toys/trunk/shab-search/tools/shab_daily_update.sh

The script fetches ItemStorage changes since the watermark, then runs shab_apply_update.py. That step:

Apply is skipped when fetch reports no new items. One job at a time (lock file). Do not start a second apply while one is running. Logs: logs/daily_update.log and logs/daily_update-YYYY-MM-DD.log.

Environment: SHAB_DAILY_SKIP_DB=1 writes JSONL only. SHAB_DAILY_ZEFIX=1 uses Zefix REST for LINDAS misses.

Full rebuild

Use this when JSONL and Postgres have drifted, or after a first load:

python tools/shab_fetch.py adopt    # or full
python tools/shab_parse.py
python tools/shab_master_companies.py run
python tools/shab_load.py --truncate --from-mastered

Once per database: sql/add_shab_entities_key_unique.sql (required for upsert), sql/add_company_lindas_fields.sql, and sql/add_company_names.sql. Search indexes: python apply_indexes.py.

Company metadata (LINDAS)

LINDAS supplies CHE↔CH-ID aliases, purpose, legal form, seat, EHRAID, street address, and other registered names (schema:legalName plus language-tagged schema:name). Build or refresh the local snapshot with python tools/shab_company_aliases.py build. --re-lindas rewrites company_lindas.jsonl (hours; required for names on the full corpus). update-keys --refresh CHE-… refreshes one firm.

Daily apply already refreshes LINDAS for companies in that day’s notices and writes both mastered JSONL and Postgres for those keys.

To copy LINDAS streets onto the whole table (or mastered JSONL) without waiting for each firm to appear in an update:

python tools/shab_backfill_lindas_addresses.py
python tools/shab_backfill_lindas_addresses.py --jsonl --skip-db

Registered names (after sql/add_company_names.sql and a LINDAS refresh that stores names):

python tools/shab_backfill_lindas_names.py --keys CHE-105.940.833 --fetch --jsonl
python tools/shab_backfill_lindas_names.py --from-jsonl

Re-running the Postgres address backfill after it has succeeded reports updated=0. Firms with no LINDAS record keep the SHAB place name only. Full-corpus names need --re-lindas first; until then only refreshed keys (daily apply or --fetch) get translations.

Operations notes

API reference in the GUI: REST API.