Why dictionaries

The problemThe table "country → shipping zone" lives as a constant in code, and the list of countries with cash on delivery lives in the next one. Every new country is a ticket, a review and a release, and the second list gets forgotten. The data changes on the business's schedule and ships on the release schedule.

Reference data often starts as a constant: a dozen lines someone typed into the code at launch. A constant is convenient until the data starts to change. Then every change goes through a developer and a release, although the decision is not a developer’s to make.

A dictionary in the admin panel separates the two schedules: the data changes when the business needs it, the code when the code needs it. But the move to the admin panel guarantees nothing by itself. How reliable a dictionary is depends on how the service reads it.

Dictionary, config and feature flag

All three take something out of code, but they answer different questions.

  • Config is how the environment is set up: addresses, timeouts, pool sizes. It differs between staging and production and changes with a deploy.
  • A feature flag is whether a behaviour is on. It lives briefly: turned on, verified, removed together with the old code branch.
  • A dictionary is facts about the domain: codes, mappings, categories. It is the same in every environment, lives for years and belongs to whoever is responsible for those facts.

If behaviour switches start going into a dictionary, that is a feature flag without an end date. If addresses and timeouts do, that is config without a deploy.

Rows are not deleted

A country you no longer ship to is tempting to just delete from the dictionary. But old orders refer to its code: purchase history, returns, last year’s reports. A deleted row makes their code unknown, and everything said in the step about an unknown value starts firing on the past.

A row is closed instead: with an active field or a validity period valid_from/valid_to. Checkout takes only active rows; history and reports take all of them, closed ones included. As a bonus a closed code stays taken, and no other country can appear under it.

If it matters what a row meant at the time of the order — which zone, which price — a flag is not enough: the values in the row get rewritten. Then either keep versions of the row with validity periods, or copy the fields you need into the order itself at checkout.

When a dictionary does not help

If the data changes together with the code that reads it, a dictionary gives nothing. A new shipping zone can be added in the admin as a row, but without code that prices it, it means nothing. What is worth moving out is what the code uses as is, without a branch of its own for every value.

And a dictionary is not the domain’s database. Thousands of records the service writes itself, relations between them and transactions are the service’s tables, not a lookup in the admin panel.

  1. Step 1 of 8

    Data that changes with a release

    A customer checks out, and the form sends the country from the address: country: CZ. Checkout needs two facts from it: which shipping zone the country falls into and whether cash on delivery is available there.

    Both facts lived in code. Zones in a forty-line map, cash on delivery in a separate list, because once it worked in two countries. The file's history is nothing but "ship to …": every new country logistics signed up meant a ticket, a review and a release.

    Worse than the slow path are two places for one decision. Cash on delivery launched in Czechia, logistics sent the announcement, the country had been in the zones for months, and nobody touched the list: CODAllowed("CZ") answers false.

    checkout/countries.go
    var shippingZones = map[string]string{
    	"DE": "eu",
    	"FR": "eu",
    	"PL": "eu",
    	"CZ": "eu",
    	// thirty more countries
    	"NO": "intl",
    	"CH": "intl",
    	"GE": "remote",
    }
    
    // cash on delivery once worked in two countries
    var codCountries = []string{"DE", "PL"}
    
    func CODAllowed(country string) bool {
    	return slices.Contains(codCountries, country)
    }

    Output

    $ git log --oneline -- checkout/countries.go
    a41c9e0 ship to Georgia
    7d2b513 move Norway to the intl zone
    c90f4aa ship to Czechia
    5e61d07 ship to Slovakia
    2f7a8c1 ship to Poland
    
    cash on delivery launched in Czechia: CODAllowed("CZ") == false
  2. Step 2 of 8

    A dictionary is data with an owner

    A dictionary is a table in the admin panel with declared fields: each has a type, whether it is required and, for enums, the allowed values. A dictionary is named by its slug, and the service requests it by that.

    Countries are added by those who arrange the shipping — logistics. They do it with a row in the admin panel, without a developer. Both facts about a country now sit in one row: the codCountries list became the cod field, and it is hard to forget when adding a country — the whole row is in front of you.

    Candidates for a dictionary share three signs: there is little data, it changes rarely and by a business decision, and it is read all the time. Shipping countries, order cancellation reasons, product categories, mappings from one set of codes to another.

    admin › dictionaries › shipping_countries
    {
      "slug": "shipping_countries",
      "fields": [
        { "key": "country", "type": "string", "required": true, "pattern": "^[A-Z]{2}$" },
        { "key": "zone",    "type": "enum",   "required": true, "values": ["eu", "intl", "remote"] },
        { "key": "cod",     "type": "bool",   "default": false }
      ],
      "items": [
        { "country": "DE", "zone": "eu",     "cod": true },
        { "country": "CZ", "zone": "eu",     "cod": true },
        { "country": "NO", "zone": "intl",   "cod": false },
        { "country": "GE", "zone": "remote", "cod": false }
      ]
    }
  3. Step 3 of 8

    How the dictionary is stored in the admin

    Three tables sit under the dictionaries in the admin. dictionaries is the dictionaries themselves: slug, title and version. dictionary_fields is the schema: one row per field, with its type, whether it is required and the allowed values. dictionary_items is the data: the whole dictionary row sits in data, and the admin checks it against the schema on every write.

    The schema is stored as rows rather than as a table per dictionary, because dictionaries are created in the admin, not by a migration. A new dictionary is a few rows in dictionary_fields, with no deploy.

    The key is pulled out of data into its own column and, together with dictionary_id, forms the primary key. A duplicate country will not get past the database. A closed row stays in the table with active = false and keeps holding its key — no other country can appear under it.

    version grows with every edit of the dictionary. From it the service can tell that nothing has changed since last time and skip fetching the batch.

    admin/migrations/001_dictionaries.sql
    CREATE TABLE dictionaries (
      id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
      slug        text NOT NULL UNIQUE,
      title       text NOT NULL,
      version     bigint NOT NULL DEFAULT 1,
      updated_at  timestamptz NOT NULL DEFAULT now()
    );
    
    -- the dictionary schema is rows, not a table: a new dictionary needs no migration
    CREATE TABLE dictionary_fields (
      dictionary_id  bigint NOT NULL REFERENCES dictionaries (id),
      key            text NOT NULL,
      type           text NOT NULL CHECK (type IN ('int', 'string', 'bool', 'enum')),
      required       boolean NOT NULL DEFAULT false,
      enum_values    text[],
      default_value  jsonb,
      is_key         boolean NOT NULL DEFAULT false,
      position       int NOT NULL,
      PRIMARY KEY (dictionary_id, key)
    );
    
    -- a dictionary has exactly one key field
    CREATE UNIQUE INDEX ON dictionary_fields (dictionary_id) WHERE is_key;
    
    CREATE TABLE dictionary_items (
      dictionary_id  bigint NOT NULL REFERENCES dictionaries (id),
      key            text NOT NULL,
      data           jsonb NOT NULL,
      active         boolean NOT NULL DEFAULT true,
      updated_at     timestamptz NOT NULL DEFAULT now(),
      updated_by     text NOT NULL,
      -- a closed row stays and keeps its key: a duplicate or a reused code will not pass
      PRIMARY KEY (dictionary_id, key)
    );
  4. Step 4 of 8

    The key is what arrives in the request

    The first decision about a dictionary is its key. For people the country name is natural, and it looks better in the admin. But the form sends the ISO code, and that is what the service will look up by.

    And names change. In 2016 the Czech Republic got the short name Czechia; in 2022 the UN started writing Turkey as Türkiye. A key by name turns a text edit in the admin into a country vanishing from checkout. Let the name be a field for people and the code the key.

    Codes are not eternal either, they just change less often. CS in ISO 3166 was the code of Czechoslovakia and, from 2003 to 2006, of Serbia and Montenegro. So the key of a row once issued is never deleted or handed to another row: the section after the walkthrough covers that.

    The second decision is whether to check a row again when the admin has already checked it against the schema. It should: the dictionary schema in the admin and the Zone type in code are two descriptions of the same thing, and they change separately. Someone adds islands to the zone enum in the admin before the code can price shipping to islands. The service should reject that row on load, not show it to a customer.

    checkout/countries/countries.go
    const Slug = "shipping_countries"
    
    var isoCode = regexp.MustCompile(`^[A-Z]{2}$`)
    
    type Country struct {
    	// the key is the ISO code from the address, not the country name
    	Code string `json:"country"`
    	Zone Zone   `json:"zone"`
    	COD  bool   `json:"cod"`
    }
    
    func (c Country) validate() error {
    	if !isoCode.MatchString(c.Code) {
    		return fmt.Errorf("%w: %q", ErrBadCode, c.Code)
    	}
    	// the admin schema can grow before the code does, so the zone is checked again
    	if !c.Zone.Valid() {
    		return fmt.Errorf("%s: %w: %q", c.Code, ErrBadZone, c.Zone)
    	}
    	return nil
    }
  5. Step 5 of 8

    The admin panel is off the request path

    The most important decision: a request never goes to the admin panel. The service keeps an in-memory copy of the dictionary, loads it on start and refreshes it in the background every few minutes. Checkout reads only the copy.

    Otherwise the admin panel — an internal tool nobody designed for checkout traffic — becomes a dependency of checkout. Its outage, a slow answer or a deploy would turn into abandoned carts.

    With a copy the admin panel can be down for an hour: the refresh logs a warning, the service answers from the last good version. The price is delay: a new row arrives not at once but within the refresh interval. Five minutes against days for a release is a good trade, but it is worth saying out loud to logistics.

    Start-up is a separate question. If the admin panel is unavailable at start, the service can come up with an empty dictionary and load it later — shipping nowhere until then. Or it can refuse to start without it. The choice depends on what is worse for this data.

    checkout/countries/store.go
    func (s *Store) Run(ctx context.Context, every time.Duration) {
    	// first load, then refresh in the background
    	s.reload(ctx)
    
    	t := time.NewTicker(every)
    	defer t.Stop()
    	for {
    		select {
    		case <-ctx.Done():
    			return
    		case <-t.C:
    			s.reload(ctx)
    		}
    	}
    }
    
    func (s *Store) reload(ctx context.Context) {
    	items, err := dictionary.Fetch[Country](ctx, s.admin, Slug)
    	if err == nil {
    		err = s.replace(items)
    	}
    	if err != nil {
    		slog.Warn("shipping_countries: keeping previous copy", "err", err)
    		s.reloadFailures.Inc()
    	}
    }

    Output

    level=INFO msg="shipping_countries loaded" items=41
    level=WARN msg="shipping_countries: keeping previous copy" err="admin: 502 Bad Gateway"
    level=INFO msg="checkout" orders=312 errors=0
  6. Step 6 of 8

    The whole batch or nothing

    The schema and the primary key in the admin check every row, but not what the code expects. Someone added islands to the zone enum for the Faroe Islands, and the code does not know that zone yet. The service should not take the dictionary at its word.

    The new version is built aside, in a separate map. Every row is validated, and duplicate keys are caught too: the admin database will not let them in, but the service need not know how it is built. If even one row fails, the whole batch is rejected and the previous copy stays in service. A partly applied dictionary is worse than the old one: nobody knows which of it is right.

    If everything passes, the new map replaces the old one with a single pointer write. Requests reading the dictionary at that moment see either the old map whole or the new one — and without locks on reads.

    The failed-refresh counter is worth an alert. The in-memory copy protects customers, but it will not tell logistics their edit did not apply.

    checkout/countries/store.go
    type Store struct {
    	current atomic.Pointer[map[string]Country]
    	// ...
    }
    
    // the whole batch or nothing: on error the dictionary is not changed
    func (s *Store) replace(items []Country) error {
    	next := make(map[string]Country, len(items))
    	for _, c := range items {
    		if err := c.validate(); err != nil {
    			return err
    		}
    		if _, dup := next[c.Code]; dup {
    			return fmt.Errorf("%w: %s", ErrDuplicate, c.Code)
    		}
    		next[c.Code] = c
    	}
    	// one pointer write: readers see the old map or the new one, never half
    	s.current.Store(&next)
    	return nil
    }

    Output

    level=WARN msg="shipping_countries: keeping previous copy" err="FO: unknown zone: \"islands\""
  7. Step 7 of 8

    An unknown value

    A customer picks Kosovo — XK. The code is not in the ISO standard, but many systems use it, and the dictionary does not have it yet. What does checkout do?

    The answer depends on the field, and here two fields get different answers. Without a zone there is no shipping price — nothing to show, and it is more honest not to offer the country than to guess. Cash on delivery for an unmarked row is simply off: the zero value false is the safe option here.

    That is why Lookup returns a second result — found or not — instead of silently handing back an empty struct: the decision about an unknown key belongs to whoever knows what it costs. Unknown codes are worth counting as a metric — that is exactly the list of rows the dictionary is missing.

    Note that the delivery options for each zone stayed in code, zoneOptions. A new zone is new logic, and a row in the admin cannot add it.

    checkout/delivery.go
    func (s *Store) Lookup(code string) (Country, bool) {
    	m := s.current.Load()
    	if m == nil {
    		return Country{}, false
    	}
    	c, ok := (*m)[code]
    	return c, ok
    }
    
    func (s *Service) DeliveryOptions(country string) ([]Option, error) {
    	c, ok := s.countries.Lookup(country)
    	if !ok {
    		s.unknownCountries.WithLabelValues(country).Inc()
    		// no zone, no shipping price: the country is not offered
    		return nil, ErrNotShippable
    	}
    
    	opts := zoneOptions[c.Zone]
    	// not marked means no cash on delivery
    	if c.COD {
    		opts = append(opts, CashOnDelivery)
    	}
    	return opts, nil
    }
  8. Step 8 of 8

    What is shared and what is each one's own

    When a service gets its first dictionary, it is tempting to build a shared cache for all future ones right away: key and validation as parameters, a "give me everything" method, a timestamp, its own set of errors. Hundreds of lines — for one dictionary.

    Only delivery is worth sharing: fetch the batch by slug and decode it into the right type. The rest — what the key is, how to validate, what to answer for an unknown value — belongs to each dictionary. Countries refuse an unknown code; a cancellation-reasons dictionary may show "other" for an unknown reason. That meaning lives next to whoever reads the dictionary: in the checkout package.

    Generalising the cache is worth it when a second dictionary appears and shows what they really share.

    dictionary/fetch.go
    // the shared part is delivery only: fetch the batch by slug and decode it into T
    func Fetch[T any](ctx context.Context, admin *Client, slug string) ([]T, error) {
    	url := admin.base + "/dictionaries/" + slug + "/items"
    	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    	if err != nil {
    		return nil, err
    	}
    
    	resp, err := admin.http.Do(req)
    	if err != nil {
    		return nil, fmt.Errorf("%s: %w", slug, err)
    	}
    	defer resp.Body.Close()
    	if resp.StatusCode != http.StatusOK {
    		return nil, fmt.Errorf("%s: admin: %s", slug, resp.Status)
    	}
    
    	var body struct {
    		Items []T `json:"items"`
    	}
    	if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
    		return nil, fmt.Errorf("%s: decode: %w", slug, err)
    	}
    	return body.Items, nil
    }

Use ← and → to move between steps.

Check yourself

Which sign says the data should move from code to a dictionary?

The admin panel is unavailable for ten minutes. What happens to checkout if the dictionary is built as in the lesson?

In a forty-country dictionary one row has a zone the code does not know. What does the service do on refresh?

Why is the dictionary keyed by the country code rather than its name?

Answered 0 of 4