Antes de propor qualquer sistema, passamos um dia no escritório da consultoria. Entrevistamos o fundador, a responsável administrativa que controla as exigências há 24 anos, o geólogo do fluxo minerário e o coordenador ambiental. Não para cumprir uma etapa de diagnóstico: o que aparece em uma reunião não é a operação inteira.

A imersão mostrou que a rotina reunia o conhecimento de poucas pessoas-chave. Automatizar apenas uma etapa preservaria a concentração de contexto. Por isso, o trabalho começou pelo mapa da operação inteira.

Mapeamos sete fluxos operacionais, agrupamos nove dores e nomeamos quatro problemas centrais. Isso virou um documento de decisão: diagnóstico, frentes priorizadas, o que fica pra depois. Liderança e equipe passaram a discutir a operação com os mesmos nomes.

A primeira entrega saiu antes de o plano fechar. Um monitor regulatório lê o Diário Oficial e o SEI todos os dias e cruza as publicações com os processos da consultoria. Quando a exigência aparece primeiro no SEI, o time ganha tempo antes da publicação no DOU. Em mineração, um prazo perdido pode parar a operação de um cliente.

A experiência da migração anterior orientou uma transição em camadas: o sistema sugere, uma pessoa confirma e a autonomia cresce conforme a equipe valida a ferramenta nova.

Na conversa de encerramento, o fundador resumiu o resultado que buscava: mais tempo para visitar clientes. Automatizar foi o gatilho do contato. O trabalho é mudar a operação em fases, sem parar o que está rodando.

Só que os dois têm relógios diferentes. Treinar e validar o modelo leva tempo. O marketplace começa a ensinar alguma coisa quando compradores reais respondem. Amarrar um ao outro significaria passar meses sem aprender sobre o mercado.

Dividimos o trabalho em duas frentes independentes. Uma ficou com o sistema de análise de qualidade, usando os dispositivos OAK-1 e o pipeline de IA. A outra ficou com o marketplace, que saiu com fotos enviadas manualmente pelos fundadores. Assim, os compradores puderam testar a vitrine antes de o modelo ficar pronto.

Essa separação levou a pergunta real a campo cedo. Passamos a primeira semana dentro do frigorífico, acompanhando o carregamento de caminhões, a escolha de carcaças e a gestão dos pedidos. Foi de lá, não do briefing, que saiu a decisão principal: um marketplace aberto reduziria o controle do frigorífico sobre a composição do estoque. O modelo virou estoque controlado.

A Bleu conduziu a validação de mercado enquanto os fundadores cuidavam da captação. Testamos a vitrine com açougues e supermercados, e os testes de usabilidade mudaram os fluxos antes da construção completa.

O piloto entrou no ar em um frigorífico real, com vendedores e compradores reais, enquanto o modelo ainda era treinado. O produto abriu conversas com 26 frigoríficos no pipeline da incubadora; esse número representa oportunidade comercial, não adoção concluída. Quando a IA ficar pronta, ela entrará em um produto ao qual o mercado já respondeu.

Quando uma parte do produto segue o prazo da pesquisa e outra precisa responder ao mercado, separar as duas não é gambiarra. É o que deixa a empresa apresentar evidência em vez de tese.

Nossa parceria mais longa, mais de 30 meses com o CoW Protocol, funciona porque essa troca não aconteceu. Nesse período saíram 16 iniciativas: SDKs, re-arquitetura de frontend, ferramentas de AMM, dados de governança. E o time principal seguiu concentrado no protocolo.

Olhando pra trás, três práticas explicam por que isso não virou carga de gestão.

A primeira: pegar o problema, não o ticket. As iniciativas chegam com pouca especificação. A gente monta o plano, valida a direção e toca. O custo de escrever a especificação completa é exatamente o custo que o time quer evitar quando delega.

A segunda: decidir dentro do contexto, não em abstrato. Re-arquitetar o SDK TypeScript ao redor do Viem, com adaptadores pra Ethers v5, v6, Viem e Wagmi, exigiu entender por que o ecossistema estava concentrado em um único stack e como preservar a compatibilidade em cada caminho.

A terceira: responder por iniciativas completas, não por tarefas. O AMM Deployer saiu em duas semanas porque o pacote inteiro era nosso pra resolver, e não uma lista de tickets repartida entre dois times.

O teste honesto pra qualquer relação desse tipo vem depois de alguns meses: o cliente está gastando mais ou menos tempo com a frente que delegou? No caso da CoW, a resposta aparece na rotina do time principal, que segue no protocolo em si.

This article explains how we implemented zero-downtime deployments for our Ponder indexer on Kubernetes. Ponder was already designed with this problem in mind. It ships with schema isolation, a standalone serve mode for horizontal scaling, and built-in health endpoints for orchestration. Our job was to wire those primitives into Kubernetes properly.

The Problem

Traditional deployment strategies fall short for blockchain indexers.

Rolling deployments assume new pods can start serving traffic quickly. That's not the case here. A fresh indexer might need 12+ hours to process historical blockchain data before it can respond to queries. During that time, the deployment appears stuck. Worse, if the new version has bugs, you've already terminated working pods.

Traditional blue-green deployments would require duplicating the entire database. Syncing a new database from scratch defeats the purpose of quick switchovers, and resource costs double during deployment windows.

Canary deployments split traffic between versions, but an unsynced indexer returns incomplete data. Users would get inconsistent responses depending on which pod handles their request.

We needed something different: a strategy that keeps serving traffic from the old version until the new one is fully ready, shares database resources between versions, allows the new indexer to take as long as needed to sync, and switches traffic atomically once ready.

The Solution: Schema-Based Blue-Green

Our approach uses PostgreSQL schema isolation combined with Kubernetes readiness probes to achieve zero-downtime deployments. The key insight:

Instead of duplicating databases, we duplicate schemas within the same database. Each deployment version writes to its own schema, and Kubernetes routes traffic only to pods that are ready to serve.

Architecture Overview

Two-Tier Deployment Pattern

We separate the indexer into two distinct Kubernetes deployments.

The Indexer Deployment (bleu-indexer-indexer) runs pnpm ponder start, the process that indexes blockchain data. It runs as a single replica since indexing is a sequential operation. This is where the heavy compute happens: it creates and populates its own database schema, consuming significant CPU and memory.

The API Deployment (bleu-indexer-api) runs pnpm ponder serve, a read-only HTTP server that Ponder provides specifically for this pattern. As described in their self-hosting guide, ponder serve runs the API layer without the indexing engine, so multiple replicas can operate behind a load balancer reading from the same database schema. We run 3 replicas in production.

This separation is crucial: the API tier can have multiple replicas for availability and load distribution, while the indexer runs as a single instance doing the heavy lifting. Ponder was designed with this decoupling in mind.

Implementation Details

Dynamic Schema Naming

Ponder uses PostgreSQL schemas to isolate each deployment. The target schema is controlled via the DATABASE_SCHEMA environment variable (or the --schema CLI flag). Ponder's docs suggest using Kubernetes pod names, git commit hashes, or deployment IDs. We compute it at pod startup:

# From the deployment entrypoint
export DATABASE_SCHEMA="bleu-indexer-v1.2.3-$(env | grep -E 'RPC_URL(_[0-9]+)?=' | sort | sha256sum | cut -c1-8)"
exec pnpm ponder start

The schema name combines the release name prefix (bleu-indexer), the image tag (e.g., v1.2.3), and a hash of all RPC URL environment variables. This ensures different versions never collide when we push a new release, and changing RPC providers triggers a fresh re-index (intentionally). Multiple schemas coexist in the same database; we clean up old ones periodically.

Ponder enforces a safety rule here: once an instance claims a schema via ponder start, no other instance can use it, even after the original stops. This prevents data corruption during concurrent deployments. It also means crash recovery works automatically: restarting a pod with the same schema resumes indexing from the last checkpoint instead of starting over.

The Magic: Readiness Probes as Traffic Gates

The key to zero-downtime is in the readiness probe configuration:

readinessProbe:
  httpGet:
    path: /ready
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 129600 # ~36 hours

That failureThreshold: 129600 is not a typo.

A new indexer might need 12+ hours to process historical blockchain data. During that time, pods stay "not ready" and Kubernetes won't route traffic to them. The previous version continues handling all requests. Once /ready returns 200, Kubernetes adds the pod to the Service endpoints and traffic shifts automatically. No manual intervention, no deployment scripts.

Ponder ships with two health endpoints designed for exactly this orchestration: /health returns 200 immediately on process startup, while /ready returns 200 only when indexing has reached realtime across all chains, returning 503 during the backfill. We didn't build custom health checks; we just pointed Kubernetes at what Ponder already provides.

Liveness vs Readiness

We use both probes with different purposes:

livenessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 129600

The liveness probe (/health) answers "Is the process alive?" If it fails three times, Kubernetes restarts the pod. The readiness probe (/ready) answers "Can this pod serve traffic?" If it fails, Kubernetes removes the pod from the Service endpoints.

A pod can be alive but not ready. That's the syncing state, and it's perfectly fine.

Deployment Flow

Here's what happens when we deploy a new indexer version:

Database Architecture

Ponder requires PostgreSQL and recommends keeping database roundtrip latency under 50ms. We use CloudNativePG to run PostgreSQL directly inside the Kubernetes cluster with a 3-replica cluster:

# Simplified from our Helm template
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: bleu-indexer-postgres
spec:
  instances: 3
  postgresql:
    parameters:
      shared_buffers: "256MB"
      max_connections: "200"
  storage:
    size: 10Gi

All schemas live in the same database. Connection pools, memory, and storage are shared across versions. Ponder also maintains a shared ponder_sync schema that caches RPC requests across instances (it's lock-free, so multiple deployments can safely share it). Old indexing schemas stick around for easy rollback if needed. We just point the API back to a previous schema. Cleanup happens as a maintenance task: periodically drop schemas that are no longer in use.

Configuration

We use Helm with environment-specific value files. The base values.yaml defines defaults: 1 indexer replica, 2 API replicas, modest resource requests. Production overrides in instances/vultr1.prod.yaml bump resources and enable ingress:

# values.yaml
indexer:
  replicas: 1
  resources:
    requests:
      cpu: 500m
      memory: 512Mi

api:
  replicas: 2
  resources:
    requests:
      cpu: 500m
      memory: 512Mi

postgres:
  instances: 3
  storage:
    size: 10Gi

# -- production overrides `instances/vultr1.prod.yaml`
indexer:
  resources:
    requests:
      cpu: 1000m
      memory: 2Gi
    limits:
      memory: 4Gi

ingress:
  enabled: true
  hosts:
    - host: api-v3.bleu.builders
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: bleu-indexer-tls
      hosts:
        - api-v3.bleu.builders

Sensitive configuration like RPC URLs and API keys live in Kubernetes Secrets, referenced via envFrom. We use Stakater Reloader to automatically restart pods when secrets change. Just annotate the deployment with reloader.stakater.com/auto: "true".

Why This Works

The old version serves traffic until the new one is fully ready (true zero-downtime). No duplicate infrastructure needed, so it's cost efficient. Kubernetes handles all the orchestration; we don't maintain custom deployment scripts. Everything lives in version control (GitOps), and we get standard Kubernetes metrics and logs for observability.

Monitoring

We track deployments through Kubernetes events for pod lifecycle, Prometheus metrics for indexer sync progress and API latency, and BetterStack for external uptime monitoring:

resource "betteruptime_monitor" "bleu_indexer" {
  url             = "https://api-v3.bleu.builders/ready"
  monitor_type    = "status"
  check_frequency = 60
}

What We Learned

Readiness probes are your traffic switch—configure them carefully. That 36-hour failure threshold seemed aggressive at first, but it's exactly what we needed.

Separating indexing from serving was the right call. They have fundamentally different scaling needs: one is a single stateful process, the other is stateless and horizontally scalable.

Schema isolation beats database duplication. It's simpler, cheaper, and gives us easy rollback for free.

Let Kubernetes do the orchestration. We tried building custom deployment scripts early on. They were fragile and hard to maintain. The readiness probe approach is declarative and self-healing.

Plan for long syncs. Blockchain history only grows. What takes 4 hours today will take 8 hours next year.

Conclusion

Zero-downtime deployments for blockchain indexers don't require complex custom tooling. By combining schema-based isolation, Kubernetes readiness probes, and a two-tier architecture, we get reliable automated deployments where users never see downtime, even when the new indexer takes hours to synchronize.

The key insight is treating the indexer and API as separate concerns with different lifecycles. The indexer is a stateful, slow-starting process. The API is stateless and can scale horizontally. By separating them and using schemas for isolation, we get the benefits of blue-green deployments without the infrastructure overhead.

References

• Ponder — Self-Hosting in Production — schema isolation, ponder serve, health endpoints, and the views pattern
• Ponder — Database Configuration — schema naming, ponder_sync cache, build ID recovery, and lifecycle rules
• Kubernetes — Configure Liveness, Readiness and Startup Probes — how readiness probes control traffic routing
• Kubernetes — Pod Lifecycle — probe types and their effect on Service endpoints
• CloudNativePG — PostgreSQL operator for Kubernetes
• Stakater Reloader — auto-restart pods on Secret/ConfigMap changes
• ArgoCD — GitOps continuous delivery for Kubernetes

You might wonder: why create another template engine when we already have excellent solutions like ERB, HAML, or Slim? I wouldn't recommend abandoning these tools for general web development. However, if you're already using Postmark templates, mustachio-ruby can help reduce your dependency on Postmark's service.

At Bleu, most of our projects use Postmark for transactional emails. We love their service and build our templates directly in their platform. However, in 2024, Postmark experienced an incident where emails took significantly longer to deliver. OTP codes were delayed by 40+ minutes. While rare, this incident made us realize we couldn't be completely dependent on a single email service. We needed a fallback system using an alternative SMTP provider like Resend.

The Problem: Template Lock-in

The quick solution was to create a fallback system that used Resend to send emails:

module FallbackMailing
  extend ActiveSupport::Concern

  included do
    if Rails.application.config.use_fallback_mailer_service
      def template_model=(model)
        @template_model = model
      end

      def template_model
        @template_model
      end
    end
  end
end

class ApplicationMailer < ActionMailer::Base
  default from: "no-reply@example.com"

  if Rails.application.config.use_fallback_mailer_service
    include FallbackMailing
  else
    include PostmarkRails::TemplatedMailerMixin

    def mail(headers = {}, &block)
      # remove not allowed attributes when using Postmark
      %i[subject header body content_type].each { |key| headers.delete(key) }
      super
    end
  end
end

This solution allowed us to send emails through alternative SMTP providers, but we faced another challenge: our templates were stored exclusively in Postmark.

We didn't want to abandon Postmark or maintain duplicate template versions. Our solution was to sync templates—keeping identical copies of our Postmark templates in our codebase. However, we discovered that Postmark's template engine was written in C#, not Ruby. This led us to create mustachio-ruby: a Ruby port of the original Mustachio template engine.

Installation

Add this line to your application's Gemfile:

gem 'mustachio-ruby'

And then execute:

bundle install

How mustachio-ruby Works

Core Components

The template engine consists of four main components that work together to parse, compile, and render templates:

1. Tokenizer (lib/mustachio_ruby/tokenizer.rb)

Parses raw template strings into structured tokens using regex patterns to identify template elements like variables, conditionals, and loops.

2. Parser (lib/mustachio_ruby/parser.rb)

Transforms tokens into executable template functions that can be called with data.

3. Context Object (lib/mustachio_ruby/context_object.rb)

Manages data access during rendering, including:

• Dot-notation access (user.profile.name)
• Parent scope access (../)
• Existence checks (exists? method)

4. Token Types (lib/mustachio_ruby/token_tuple.rb)

Defines template element types: HTML-escaped variables, raw variables, conditional blocks, array iteration, and negated conditionals.

Basic Usage

Simple Variable Interpolation

template = MustachioRuby.parse("Hello {{name}}!")
content = template.call({"name" => "World"})
# => "Hello World!"

Array Iteration

template = "{{#each Company.ceo.products}}<li>{{ name }} and {{version}} and has a CEO: {{../../last_name}}</li>{{/each}}"
renderer = MustachioRuby.parse(template)

model = {
  "Company" => {
    "ceo" => {
      "last_name" => "Smith",
      "products" => [
        { "name" => "name 0", "version" => "version 0" },
        { "name" => "name 1", "version" => "version 1" },
      ]
    }
  }
}

result = renderer.call(model)
# =>
# <li>name 0 and version 0 and has a CEO: Smith</li>
# <li>name 1 and version 1 and has a CEO: Smith</li>

Conditional Rendering

# Show content when user exists
template = MustachioRuby.parse("{{#user}}Welcome {{name}}!{{/user}}")

# Show content when user doesn't exist
template = MustachioRuby.parse("{{^user}}Please log in{{/user}}")

Configuration Options

options = MustachioRuby::ParsingOptions.new
options.disable_content_safety = true # Disable HTML escaping
options.source_name = "my_template" # For error reporting
options.token_expanders = [expander] # Custom extensions

template = MustachioRuby.parse(source, options)

HTML Email Templates

You can follow the Postmark template syntax to build your email templates and then use mustachio-ruby (or the original C# version) to interpolate your variables:

mustachio_template = <<~HTML
  <div>
    <h1>Hello {{name}}!</h1>
    {{#user}}
      <p>Welcome back!</p>
    {{/user}}
    {{^user}} <!-- negative check -->
      <p>Please log in</p>
    {{/user}}
    <ul>
      {{#each items}}
        <li>{{name}}: ${{price}}</li>
      {{/each}}
    </ul>
  </div>
HTML

data = {
  "name" => "John",
  "user" => { "name" => "John" },
  "items" => [
    { "name" => "Apple", "price" => 1.50 },
    { "name" => "Banana", "price" => 0.75 }
  ]
}

template = MustachioRuby.parse(mustachio_template)
result = template.call(data)

# =>
# <div>
#   <h1>Hello John!</h1>
#   <p>Welcome back!</p>
#   <ul>
#     <li>Apple: $1.5</li>
#     <li>Banana: $0.75</li>
#   </ul>
# </div>

Ruby on Rails Integration

To use mustachio-ruby in Rails, set the body to be the parsed HTML content and the mail content_type to be "text/html" (or "text/plain" if you're using mustachio to interpolate variables in a plain text string):

def mail(headers = {}, &)
  template_content = get_template(template_alias)
  options = MustachioRuby::ParsingOptions.new
  options.disable_content_safety = true
  renderer = MustachioRuby.parse(template_content, options)
  html_content = renderer.call(template_model.deep_stringify_keys)
  headers[:body] = html_content
  headers[:content_type] = "text/html"

  super
end

Final Thoughts

Postmark is a fantastic service, and we continue to use it every day. mustachio-ruby isn't a replacement—it's a safety net. It lets you render templates in pure Ruby and send emails even when you don't have access to Postmark.

If your team relies on Postmark templates, I hope mustachio-ruby gives you the flexibility to build resilient email systems without vendor lock-in.

Learn More

• mustachio-ruby on GitHub
• Postmark Template Syntax Documentation
• Original Mustachio (C#) Repository