Creating config
Developing a DipDup indexer begins with creating a YAML config file. You can find a minimal example to start indexing on the Quickstart page.
General structure
DipDup configuration is stored in YAML files of a specific format. By default, DipDup searches for dipdup.yml
file in the current working directory, but you can provide any path with a -c
CLI option.
DipDup config file consists of several logical blocks:
Header | spec_version * | 14.15. spec_version |
package * | 14.12. package | |
Inventory | database | 14.5. database |
contracts | 14.3. contracts | |
datasources | 14.6. datasources | |
custom | 14.4. custom | |
Index definitions | indexes | 14.9. indexes |
templates | 14.16. templates | |
Hook definitions | hooks | 14.8. hooks |
jobs | 14.10. jobs | |
Integrations | hasura | 14.7. hasura |
sentry | 14.14. sentry | |
prometheus | 14.13. prometheus | |
Tunables | advanced | 14.2. advanced |
logging | 14.11. logging |
Header contains two required fields, package
and spec_version
. They are used to identify the project and the version of the DipDup specification. All other fields in the config are optional.
Inventory specifies contracts that need to be indexed, datasources to fetch data from, and the database to store data in.
Index definitions define the index templates that will be used to index the contract data.
Hook definitions define callback functions that will be called manually or on schedule.
Integrations are used to integrate with third-party services.
Tunables affect the behavior of the whole framework.
Merging config files
DipDup allows you to customize the configuration for a specific environment or workflow. It works similarly to docker-compose anchors but only for top-level sections. If you want to override a nested property, you need to recreate a whole top-level section. To merge several DipDup config files, provide the -c
command-line option multiple times:
dipdup -c dipdup.yml -c dipdup.prod.yml run
Run config export
command if unsure about the final config used by DipDup.
Full example
This page or paragraph is yet to be written. Come back later.
Let's put it all together. The config below is an artificial example but contains almost all available options.
spec_version: 1.2
package: my_indexer
database:
kind: postgres
host: db
port: 5432
user: dipdup
password: changeme
database: dipdup
schema_name: public
immune_tables:
- token_metadata
- contract_metadata
contracts:
some_dex:
address: KT1K4EwTpbvYN9agJdjpyJm4ZZdhpUNKB3F6
typename: quipu_fa12
datasources:
tzkt_mainnet:
kind: tzkt
url: https://api.tzkt.io
my_api:
kind: http
url: https://my_api.local/v1
ipfs:
kind: ipfs
url: https://ipfs.io/ipfs
coinbase:
kind: coinbase
metadata:
kind: metadata
url: https://metadata.dipdup.net
network: mainnet
indexes:
operation_index_from_template:
template: operation_template
values:
datasource: tzkt
contract: some_dex
big_map_index_from_template:
template: big_map_template
values:
datasource: tzkt
contract: some_dex
first_level: 1
last_level: 46963
skip_history: never
factory:
kind: operation
datasource: tzkt
types:
- origination
contracts:
- some_dex
handlers:
- callback: on_factory_origination
pattern:
- type: origination
similar_to: some_dex
templates:
operation_template:
kind: operation
datasource: <datasource>
types:
- origination
- transaction
contracts:
- <contract>
handlers:
- callback: on_origination
pattern:
- type: origination
originated_contract: <contract>
- callback: on_some_call
pattern:
- type: transaction
destination: <contract>
entrypoint: some_call
big_map_template:
kind: big_map
datasource: <datasource>
handlers:
- callback: on_update_records
contract: <name_registry>
path: store.records
- callback: on_update_expiry_map
contract: <name_registry>
path: store.expiry_map
hooks:
calculate_stats:
callback: calculate_stats
atomic: False
args:
major: bool
jobs:
midnight_stats:
hook: calculate_stats
crontab: "0 0 * * *"
args:
major: True
sentry:
dsn: https://localhost
environment: dev
debug: False
prometheus:
host: 0.0.0.0
hasura:
url: http://hasura:8080
admin_secret: changeme
allow_aggregations: False
camel_case: true
select_limit: 100
advanced:
early_realtime: True
merge_subscriptions: False
postpone_jobs: False
metadata_interface: False
skip_version_check: False
scheduler:
apscheduler.job_defaults.coalesce: True
apscheduler.job_defaults.max_instances: 3
reindex:
manual: wipe
migration: exception
rollback: ignore
config_modified: exception
schema_modified: exception
rollback_depth: 2
crash_reporting: False
logging: verbose