Quickstart

This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details.

Let's create an indexer for the tzBTC FA1.2 token contract. Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.

A modern Linux/MacOS distribution with Python 3.10 installed is required to run DipDup.

Create a new project

You can initialize a hello-world project interactively by choosing configuration options in the terminal. The following command will install cookiecutter and create a new project in the current directory.

python -c "$(curl -sSL https://dipdup.net/install.py)"

From scratch

We advise using the Poetry package manager for new projects. However, it's not a requirement. If you prefer pdb, piptools, pipenv or other tools — use them instead.

# Create a new project
mkdir my-indexer; cd my-indexer
poetry init --python ">=3.10,<3.11"
# Add dipdup as a dependency
poetry add dipdup
# Enter the virtualenv
poetry shell

💡 SEE ALSO

Write a configuration file

DipDup configuration is stored in YAML files of a specific format. Create a new file named dipdup.yml in your current working directory with the following content:

spec_version: 1.2
package: demo_tzbtc

database:
  kind: sqlite
  path: demo_tzbtc.sqlite3

contracts:
  tzbtc_mainnet:
    address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
    typename: tzbtc

datasources:
  tzkt:
    kind: tzkt
    url: https://api.tzkt.io

advanced:
  early_realtime: True
  crash_reporting: True

indexes:
  tzbtc_holders_mainnet:
    kind: operation
    datasource: tzkt
    contracts:
      - tzbtc_mainnet
    handlers:
      - callback: on_transfer
        pattern:
          - destination: tzbtc_mainnet
            entrypoint: transfer
      - callback: on_mint
        pattern:
          - destination: tzbtc_mainnet
            entrypoint: mint

💡 SEE ALSO

Initialize project tree

Now it's time to generate typeclasses and callback stubs. Run the following command:

dipdup init

DipDup will create a Python package demo_tzbtc having the following structure:

demo_tzbtc
├── graphql
├── handlers
│   ├── __init__.py
│   ├── on_mint.py
│   └── on_transfer.py
├── hooks
│   ├── __init__.py
│   ├── on_reindex.py
│   ├── on_restart.py
│   ├── on_index_rollback.py
│   └── on_synchronized.py
├── __init__.py
├── models.py
├── sql
│   ├── on_reindex
│   ├── on_restart
│   ├── on_index_rollback
│   └── on_synchronized
└── types
    ├── __init__.py
    └── tzbtc
        ├── __init__.py
        ├── parameter
        │   ├── __init__.py
        │   ├── mint.py
        │   └── transfer.py
        └── storage.py

That's a lot of files and directories! But don't worry, we will need only models.py and handlers modules in this guide.

💡 SEE ALSO

Define data models

Our schema will consist of a single model Holder having several fields:

  • address — account address
  • balance — in tzBTC
  • volume — total transfer/mint amount bypassed
  • tx_count — number of transfers/mints
  • last_seen — time of the last transfer/mint

Put the following content in the models.py file:

from tortoise import fields

from dipdup.models import Model


class Holder(Model):
    address = fields.CharField(max_length=36, pk=True)
    balance = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    turnover = fields.DecimalField(decimal_places=8, max_digits=20, default=0)
    tx_count = fields.BigIntField(default=0)
    last_seen = fields.DatetimeField(null=True)

💡 SEE ALSO

Implement handlers

Everything's ready to implement an actual indexer logic.

Our task is to index all the balance updates, so we'll start with a helper method to handle them. Create a file named on_balance_update.py in the handlers package with the following content:

from datetime import datetime
from decimal import Decimal

import demo_tzbtc.models as models


async def on_balance_update(
    address: str,
    balance_update: Decimal,
    timestamp: datetime,
) -> None:
    holder, _ = await models.Holder.get_or_create(address=address)
    holder.balance += balance_update
    holder.turnover += abs(balance_update)
    holder.tx_count += 1
    holder.last_seen = timestamp
    await holder.save()

Three methods of tzBTC contract can alter token balances — transfer, mint, and burn. The last one is omitted in this tutorial for simplicity. Edit corresponding handlers to call the on_balance_update method with data from matched operations:

on_transfer.py

from decimal import Decimal

from demo_tzbtc.handlers.on_balance_update import on_balance_update
from demo_tzbtc.types.tzbtc.parameter.transfer import TransferParameter
from demo_tzbtc.types.tzbtc.storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models import Transaction


async def on_transfer(
    ctx: HandlerContext,
    transfer: Transaction[TransferParameter, TzbtcStorage],
) -> None:
    if transfer.parameter.from_ == transfer.parameter.to:
        # NOTE: Internal tzBTC transfer
        return

    amount = Decimal(transfer.parameter.value) / (10**8)
    await on_balance_update(
        address=transfer.parameter.from_,
        balance_update=-amount,
        timestamp=transfer.data.timestamp,
    )
    await on_balance_update(
        address=transfer.parameter.to,
        balance_update=amount,
        timestamp=transfer.data.timestamp,
    )

on_mint.py

from decimal import Decimal

from demo_tzbtc.handlers.on_balance_update import on_balance_update
from demo_tzbtc.types.tzbtc.parameter.mint import MintParameter
from demo_tzbtc.types.tzbtc.storage import TzbtcStorage
from dipdup.context import HandlerContext
from dipdup.models import Transaction


async def on_mint(
    ctx: HandlerContext,
    mint: Transaction[MintParameter, TzbtcStorage],
) -> None:
    amount = Decimal(mint.parameter.value) / (10**8)
    await on_balance_update(
        address=mint.parameter.to,
        balance_update=amount,
        timestamp=mint.data.timestamp,
    )

And that's all! We can run the indexer now.

💡 SEE ALSO

Run your indexer

dipdup run

DipDup will fetch all the historical data and then switch to realtime updates. Your application data has been successfully indexed!

💡 SEE ALSO