`x/storage` — schemaless storage with typed queries

The x/storage/schemaless package gives you one repository API for saving, loading, and querying Go values — including union types — without writing serialization code or SQL. You pick a backend by picking a constructor; everything else stays the same.

Installation

The package ships with the mkunion module:

go get github.com/widmogrod/mkunion

Import what you need:

import (
    "github.com/widmogrod/mkunion/x/storage/schemaless"           // repositories
    "github.com/widmogrod/mkunion/x/storage/predicate"            // typed query predicates
    "github.com/widmogrod/mkunion/x/storage/schemaless/typedful"  // type-safe wrapper
)

Storage adapters

In-memory

repo := schemaless.NewInMemoryRepository[MyRecord]()

Best for tests and local development. It also defines the full contract: every behaviour in the feature matrix below is supported.

DynamoDB

client := dynamodb.NewFromConfig(cfg)
repo := schemaless.NewDynamoDBRepository[MyRecord](client, tableName)

The table needs a string hash key ID and a string range key Type.

DynamoDB ignores Sort. Queries against this backend return matching records in an arbitrary order - see Sorting by a data field.

OpenSearch

client, _ := opensearch.NewClient(opensearch.Config{Addresses: []string{address}})
repo := schemaless.NewOpenSearchRepository[MyRecord](client, indexName)

Writes use Refresh=true, so reads observe writes immediately.

Change streams — the append log

A repository answers "what is the state now". The append log answers "what just changed": it is an ordered stream of Change[T] events that subscribers consume, one per save or delete.

type Change[T any] struct {
    Before  *Record[T] // state before the change (nil on first insert)
    After   *Record[T] // state after the change (nil on delete)
    Deleted bool       // true when the record was deleted
    Offset  int        // position in the log, increases with every change
}

The in-memory repository feeds its log automatically — every UpdateRecords appends the resulting changes:

repo := schemaless.NewInMemoryRepository[MyRecord]()
log := repo.AppendLog()

err := log.Subscribe(ctx, 0, nil, func(change schemaless.Change[MyRecord]) {
    // react to the change
})

Subscribe blocks and delivers changes until the log ends:

fromOffset — 0 starts at the beginning, -1 at the latest change, and any other value resumes from the change with that Offset (as delivered in an earlier subscription).
filter — a predicate.WherePredicates that delivers only matching changes; nil delivers everything.
Return value — nil after Close() once all changes are delivered, the context's error when the context is cancelled.

A log can also be written to directly, without a repository, via Push, Change, and Delete.

Append log implementations

Implementation	What it is
`schemaless.AppendLog[T]`	In-memory log; defines the full contract
`typedful.TypedAppendLog[T]`	Typed wrapper over an `AppendLoger[schema.Schema]`; translates records and filter predicates both ways. `Append` (merging another log) is not supported and is declared as the `MergeAppend` downgrade
`schemaless.KinesisStream`	Not an `AppendLoger` yet: it has a different `Subscribe` signature and no write side; it only reads DynamoDB change events from a Kinesis stream. Making it conform is an open follow-up

Like the repositories, every implementation runs the same behavioural specification, with explicit downgrades:

spec.RunAppendLogSpec(t, spec.AppendLogTypedful, newLog,
    spec.FullAppendLogCapabilities().
        WithoutMergeAppend(),
)

The results render as the Append log capability matrix below.

Feature matrix

Every adapter is verified against the same behavioural specification (x/storage/schemaless/spec). An adapter that cannot provide a behaviour declares the downgrade explicitly in its test wiring, for example:

spec.RunRepositorySpec(t, spec.BackendDynamoDB, newRepo,
    spec.FullCapabilities().
        WithoutSortByDataField().
        WithoutBackwardPagination(),
)

The tables below are generated by running that test suite — they are not written by hand, so they cannot drift from what the code actually does.

Repository capability matrix

Every Repository backend passes the same behavioural spec (spec.RunRepositorySpec), modulo the capabilities it explicitly downgrades. The in-memory repository defines the full contract.

Capability	In-Memory	DynamoDB	OpenSearch
SortByDataField — Sort results by any record field (e.g. `Data.Name`)	✅	⛔	✅
BackwardPagination — Page backward with `Before` cursors and `Prev` links	✅	⛔	⛔
AtomicBatch — All-or-nothing `UpdateRecords` batches	✅	✅	⛔
MonotonicOverwriteVersion — Versions keep increasing under `PolicyOverwriteServerChanges`	✅	✅	⛔

Repository verified behaviours

Each row is a spec subtest; ✅ verified, ⛔ skipped by a declared capability downgrade, ❌ failing when the report was generated.

Suite	Behaviour	In-Memory	DynamoDB	OpenSearch
repository	get of a missing record returns ErrNotFound	✅	✅	✅
repository	get with the wrong record type returns ErrNotFound	✅	✅	✅
repository	delete of a missing record is not an error	✅	✅	✅
repository	empty update command returns ErrEmptyCommand	✅	✅	✅
repository	saved record can be read back with its data	✅	✅	✅
repository	update with the current version succeeds and bumps the version	✅	✅	✅
repository	write with a stale version fails with ErrVersionConflict	✅	✅	✅
repository	PolicyOverwriteServerChanges wins over a stale version	✅	✅	✅
repository	overwrites keep the version increasing	✅	✅	⛔ `MonotonicOverwriteVersion`
repository	deleted record is gone	✅	✅	✅
repository	where predicate filters records	✅	✅	✅
repository	batch result maps key records by ID and type	✅	✅	✅
repository	where predicate accepts literal values	✅	✅	✅
repository	empty where query matches all records	✅	✅	✅
repository	OR predicate matches either branch	✅	✅	✅
repository	NOT over OR excludes both branches	✅	✅	✅
repository	query without a record type is not an error	✅	✅	✅
repository	record type separates records	✅	✅	✅
repository	forward pagination visits every record exactly once	✅	✅	✅
repository	sorting orders records by a data field	✅	⛔ `SortByDataField`	✅
repository	sorting orders records by a numeric data field	✅	⛔ `SortByDataField`	✅
repository	sorted pagination keeps order across pages	✅	⛔ `SortByDataField`	✅
repository	prev cursor pages backward	✅	⛔ `BackwardPagination`	⛔ `BackwardPagination`
repository	batch with a conflict writes nothing	✅	✅	⛔ `AtomicBatch`
complex queries	union variants survive a round trip	✅	✅	✅
complex queries	filter on a field inside a union variant	✅	✅	✅
complex queries	OR matches across union variants	✅	✅	✅
complex queries	NOT excludes a union variant match, records without the field stay	✅	✅	✅
complex queries	AND combines a plain field with a union field	✅	✅	✅
complex queries	numeric equality filter	✅	✅	✅
complex queries	string equality filter	✅	✅	✅
complex queries	filter on a nonexistent field returns no records and no error	✅	✅	✅
complex queries	update of a union variant is queryable afterwards	✅	✅	✅

Append log capability matrix

Every AppendLoger implementation passes the same behavioural spec (spec.RunAppendLogSpec), modulo the capabilities it explicitly downgrades. The in-memory append log defines the full contract.

Capability	In-Memory	Typedful
Filtering — `Subscribe` honours a where-predicate filter	✅	✅
OffsetResume — `Subscribe` resumes from a given change offset	✅	✅
Replay — A late subscriber receives every past change	✅	✅
MergeAppend — `Append` merges another log's changes	✅	⛔

Append log verified behaviours

Each row is a spec subtest; ✅ verified, ⛔ skipped by a declared capability downgrade, ❌ failing when the report was generated.

Suite	Behaviour	In-Memory	Typedful
append log	pushed changes reach a subscriber in order with increasing offsets	✅	✅
append log	change and delete emit corresponding events	✅	✅
append log	every subscriber receives every change	✅	✅
append log	close unblocks a subscriber waiting on an empty log	✅	✅
append log	context cancellation unblocks a waiting subscriber	✅	✅
append log	a closed log replays every change to a late subscriber	✅	✅
append log	subscription resumes from a given offset	✅	✅
append log	filter delivers only matching changes	✅	✅
append log	append merges another log's changes	✅	⛔ `MergeAppend`

Sorting by a data field

SortByDataField is the one capability that splits the backends, so it decides which backend a sorted query belongs on.

Backend	`Sort` on `Data.#.Name`
In-Memory	works
OpenSearch	works
DynamoDB	ignored

If you need sorted queries, use OpenSearch. In-Memory is the reference implementation used by the spec suite; OpenSearch is the production backend that sorts.

Why DynamoDB cannot do it

DynamoDB only returns records in the order of a key, and this table's key is ID + Type.

FindingRecords issues a Scan, and Scan has no ordering option - the order of returned records is not defined.
Even a Query can only order by the table's range key, ascending or descending, never by an arbitrary attribute.
An index key must be a flat, top-level attribute, and Data.#.Name is nested inside Data.

The Sort value is accepted and carried across pages, but never applied. The adapter declares this honestly with WithoutSortByDataField(), so the matrix above shows a skip instead of a pass.

What supporting it would take

Not a configuration flag - a write-path change:

every write copies Data.Name into a flat top-level attribute;
a global secondary index keyed on it (hash Type, range the new attribute), with existing records rewritten so they reappear in sorted results;
index discovery at start-up (DescribeTable), so Sort is accepted for indexed fields and rejected with a clear error for the rest.

Creating that index automatically is out of scope on purpose: it costs money, needs wider IAM permissions, backfills slowly, and DynamoDB caps a table at 20 indexes. It would have to be an opt-in constructor, never a start-up guess.

Adding a new adapter

Implement schemaless.Repository[T] — or schemaless.AppendLoger[T] for a change stream.
Wire it into the spec suite next to your adapter, declaring any capability downgrades explicitly (see x/storage/schemaless/inmemory_spec_test.go for a repository, x/storage/schemaless/appendlog_spec_test.go for an append log).
Run go test ./x/storage/schemaless/. On a green run the feature matrix above and x/storage/README.md regenerate themselves — commit them with your change.

x/storage — schemaless storage with typed queries