meowlib/doc/multi_device_sync_plan.md

Multi-Device Conversation Sync — Implementation Plan

Context

meowlib already has scaffolding for multi-device sync:

Existing artefact	Where
Identity.Device *KeyPair	client/identity.go:35
Identity.OwnedDevices PeerList	client/identity.go:40
Peer.Type string	client/peer.go:52
ToServerMessage.device_messages (field 10)	pb/messages.proto:75
FromServerMessage.device_messages (field 9)	pb/messages.proto:99
BackgroundJob.Device *KeyPair	client/identity.go:334

The server (server/router.go) does not yet implement device_messages routing; it goes through messages/Chat today.

---

Chosen Sync Scheme: Event-Driven Delta Sync over Existing Message Infrastructure

Rationale

Approach	Pros	Cons	Verdict
Full DB sync	Complete history	Huge payloads, merge conflicts, wasteful	❌
Inbox/outbox file sharing	Simple to reason about	File-level granularity, no dedup, breaks privacy model	❌
Event-driven delta sync	Minimal data, no merge needed, reuses existing crypto + server stack	Requires dedup table	✅

Each message event (received, sent, status change) is forwarded immediately to sibling devices through the same server infrastructure as regular peer messages. Each device maintains its own complete local DB. Convergence is eventual; dedup via ConversationStatus.Uuid.

Key Design Decisions

1. Zero server changes required. Device sync messages are addressed to the sibling device's lookup key and travel through the existing msg:{lookup_key} Redis sorted-set on the server, returned in from_server.Chat — identical to peer messages.

2. Device peers reuse the Peer struct with Type = "device", stored in Identity.OwnedDevices. They have their own three keypairs (MyIdentity, MyEncryptionKp, MyLookupKp) and MyPullServers.

3. A new proto message DeviceSyncPayload is added to messages.proto. It is serialised and placed in UserMessage.Appdata; the parent UserMessage.Type is set to "device_sync". This lets the client recognise sync messages without any server-side awareness.

4. GetRequestJobs() is extended to include device lookup keys alongside peer lookup keys for the appropriate servers, so the background poll thread picks up device sync messages without any extra call.

5. Dedup is handled by a small SQLite table device_sync_seen (one table per identity folder, not per peer) keyed on DeviceSyncPayload.DedupId.

---

New Protobuf Message

Add to pb/messages.proto before re-generating:

// Payload carried inside UserMessage.appdata for device-to-device sync.
// The enclosing UserMessage.type MUST be "device_sync".
message DeviceSyncPayload {
    string sync_type  = 1; // "msg" | "status" | "peer_event"
    string peer_uid   = 2; // local UID of the peer conversation on the sending device
    DbMessage db_message = 3; // the DbMessage to replicate
    string dedup_id   = 4; // globally unique ID (= DbMessage.status.uuid or generated)
}

Run cd pb && ./protogen.sh after adding this.

---

Implementation Phases

Phase 1 — Device Pairing

Files to touch: client/identity.go, client/helpers/ (new file deviceHelper.go)

Goal: Allow two app instances owned by the same user to establish a shared keypair relationship, mirroring the peer invitation flow but flagging the peer as Type = "device".

1.1 Identity.InitDevicePairing(myDeviceName string, serverUids []string) (*Peer, error)

• Identical to InvitePeer but sets peer.Type = "device".
• Stores the resulting peer in Identity.OwnedDevices (not Peers).
• Returns the peer so the caller can produce a ContactCard QR/file.

1.2 Identity.AnswerDevicePairing(myDeviceName string, receivedContact *meowlib.ContactCard) (*Peer, error)

• Mirrors AnswerInvitation, stores in OwnedDevices.

1.3 Identity.FinalizeDevicePairing(receivedContact *meowlib.ContactCard) error

• Mirrors FinalizeInvitation, operates on OwnedDevices.

1.4 Helper functions (new file client/helpers/deviceHelper.go)

// DevicePairingCreateMessage – wraps an invitation step-1 for a device peer.
func DevicePairingCreateMessage(peer *client.Peer, serverUid string) ([]byte, string, error)

// DevicePairingAnswerMessage – wraps invitation step-3 answer for a device peer.
func DevicePairingAnswerMessage(peer *client.Peer, serverUid string) ([]byte, string, error)

These reuse invitationCreateHelper.go/invitationAnswerHelper.go logic.

1.5 Extend PeerStorage operations for OwnedDevices

OwnedDevices is currently a PeerList (in-memory slice). For scalability it should use the same Badger-backed PeerStorage mechanism as Peers. Consider adding a second PeerStorage field DeviceStorage to Identity with its own DbFile.

---

Phase 2 — Sync Payload Helpers

Files to touch: client/helpers/deviceHelper.go (continued), client/dbmessage.go

2.1 Build a sync message for one sibling device

// BuildDeviceSyncMessage wraps a DbMessage into a UserMessage addressed to a
// sibling device peer. The caller then calls peer.ProcessOutboundUserMessage.
func BuildDeviceSyncMessage(
    devicePeer *client.Peer,
    syncType   string,           // "msg" | "status" | "peer_event"
    peerUid    string,
    dbm        *meowlib.DbMessage,
    dedupId    string,
) (*meowlib.UserMessage, error)

Implementation:

1. Serialise DeviceSyncPayload{SyncType, PeerUid, DbMessage, DedupId} with proto.Marshal.
2. Create a UserMessage with Type = "device_sync", Destination = devicePeer.ContactLookupKey, Appdata = serialisedPayload.
3. Set Status.Uuid = dedupId.

2.2 Dispatch sync to all sibling devices

// DispatchSyncToDevices sends a DeviceSyncPayload to every device peer whose
// pull server list overlaps with the available servers.
// It enqueues a SendJob per device, reusing the existing bgSendHelper queue.
func DispatchSyncToDevices(
    storagePath string,
    syncType    string,
    peerUid     string,
    dbm         *meowlib.DbMessage,
    dedupId     string,
) error

Iterates identity.OwnedDevices, builds and queues one SendJob per device (just like CreateUserMessageAndSendJob but using device peer keys and putting the message in outbox/ with a recognisable prefix, e.g. dev_{devPeerUid}_{dedupId}).

The message is packed into ToServerMessage.Messages (same field as regular chat). No server changes needed.

---

Phase 3 — Integrate Dispatch into Send/Receive Paths

Files to touch: client/helpers/messageHelper.go, client/helpers/bgPollHelper.go

3.1 After outbound message stored (CreateAndStoreUserMessage)

At the end of CreateAndStoreUserMessage (after peer.StoreMessage), add:

// Async: do not block the caller
go DispatchSyncToDevices(storagePath, "msg", peerUid, dbm, usermessage.Status.Uuid)

The dbm is obtained from UserMessageToDbMessage(true, usermessage, nil) (files are excluded from sync — they stay on the originating device or are re-requested).

3.2 After inbound message stored (ConsumeInboxFile)

After peer.StoreMessage(usermsg, filenames) succeeds:

dbm := client.UserMessageToDbMessage(false, usermsg, nil)
go DispatchSyncToDevices(storagePath, "msg", peer.Uid, dbm, usermsg.Status.Uuid)

3.3 After ACK status update (ReadAckMessageResponse — currently a stub)

When status timestamps (received/processed) are updated in the DB, dispatch a "status" sync with the updated DbMessage.

---

Phase 4 — Receive & Consume Device Sync Messages

Files to touch: client/helpers/bgPollHelper.go, new client/helpers/deviceSyncHelper.go

4.1 Extend GetRequestJobs() to include device lookup keys

In identity.go:GetRequestJobs(), after the loop over Peers, add a similar loop over OwnedDevices:

for _, devPeer := range id.OwnedDevices {
    for _, server := range devPeer.MyPullServers {
        if job, ok := srvs[server]; ok {
            job.LookupKeys = append(job.LookupKeys, devPeer.MyLookupKp)
        }
    }
}

Device messages will now arrive inside from_server.Chat alongside regular peer messages. The next step distinguishes them.

4.2 Distinguish device vs peer messages in ConsumeInboxFile

After identity.Peers.GetFromMyLookupKey(packedUserMessage.Destination) returns nil, try:

devPeer := identity.OwnedDevices.GetFromMyLookupKey(packedUserMessage.Destination)
if devPeer != nil {
    err := ConsumeDeviceSyncMessage(devPeer, packedUserMessage)
    // continue to next message
    continue
}
// original error path

4.3 ConsumeDeviceSyncMessage (new file client/helpers/deviceSyncHelper.go)

func ConsumeDeviceSyncMessage(
    devPeer *client.Peer,
    packed  *meowlib.PackedUserMessage,
) error

Steps:

1. Decrypt with devPeer.ProcessInboundUserMessage(packed.Payload, packed.Signature).
2. Check usermsg.Type == "device_sync".
3. Deserialise DeviceSyncPayload from usermsg.Appdata.
4. Dedup check: call IsDeviceSyncSeen(payload.DedupId). If yes, skip.
5. Mark seen: MarkDeviceSyncSeen(payload.DedupId).
6. Dispatch by payload.SyncType:
   • "msg": find the local peer by payload.PeerUid, call client.StoreDeviceSyncedMessage(peer, payload.DbMessage).
   • "status": update the status fields in the existing DB row matched by payload.DbMessage.Status.Uuid.
   • "peer_event": (future) update peer metadata.

4.4 StoreDeviceSyncedMessage in client/messagestorage.go

A thin wrapper around storeMessage that:

• Marks the message as synced (a new bool field Synced in DbMessage, or use a naming convention in DbMessage.Appdata).
• Does not trigger a second round of sync dispatch (no re-broadcast).
• Handles absent file paths gracefully (files are not synced, only metadata).

---

Phase 5 — Dedup Store

Files to touch: new client/devicesyncdedup.go

A single SQLite DB per identity folder: {StoragePath}/{IdentityUuid}/devicesync.db.

Schema:

CREATE TABLE IF NOT EXISTS seen (
    id        TEXT NOT NULL PRIMARY KEY,
    seen_at   INTEGER NOT NULL
);

Functions:

func IsDeviceSyncSeen(storagePath, identityUuid, dedupId string) (bool, error)
func MarkDeviceSyncSeen(storagePath, identityUuid, dedupId string) error
func PruneDeviceSyncSeen(storagePath, identityUuid string, olderThan time.Duration) error

PruneDeviceSyncSeen is called periodically (e.g. weekly) from the background thread to remove entries older than 30 days.

---

File Change Summary

File	Change
pb/messages.proto	Add DeviceSyncPayload message
pb/protogen.sh → re-run	Regenerate .pb.go
client/identity.go	Add InitDevicePairing, AnswerDevicePairing, FinalizeDevicePairing; extend GetRequestJobs()
client/peer.go	No changes needed (Type field already exists)
client/messagestorage.go	Add StoreDeviceSyncedMessage
client/devicesyncdedup.go	New — dedup SQLite helpers
client/helpers/deviceHelper.go	New — BuildDeviceSyncMessage, DispatchSyncToDevices, pairing message helpers
client/helpers/deviceSyncHelper.go	New — ConsumeDeviceSyncMessage
client/helpers/messageHelper.go	Add DispatchSyncToDevices call after outbound store
client/helpers/bgPollHelper.go	Add device message detection in ConsumeInboxFile

Server package: no changes required.

---

Sync Scope

Data	Synced	Notes
Message text / data	✅	In DbMessage.Data
Outbound flag	✅	In DbMessage.Outbound
Message UUID	✅	Via ConversationStatus.Uuid
Sent/received timestamps	✅	In ConversationStatus
File content	❌	Not synced; only FilePaths metadata synced
Peer metadata (keys, servers)	❌	Phase 2+ scope
Identity blob	❌	Out of scope; handled by manual export

---

Privacy Properties

• Device sync messages are end-to-end encrypted (same X25519 + PGP as peer messages).
• The server sees only the device lookup key as destination; it has no knowledge this is a sync vs a peer message.
• Including device lookup keys in batch pull requests does not leak which other device belongs to you (same privacy model as multiple peer lookup keys per request).
• OwnedDevices peers should be considered "hidden" (not shown in contact lists) and can optionally be stored in the hidden peer store.

---

Testing Strategy

1. Unit tests for DeviceSyncPayload serialisation round-trip.
2. Unit tests for dedup store (seen/mark/prune lifecycle).
3. Integration test extending TestEndToEnd:
   • Create identity, two device peers (DeviceA, DeviceB).
   • Send a message on DeviceA.
   • Verify DeviceB's DB contains the synced message after ConsumeDeviceSyncMessage.
   • Resend the same dedup_id — verify no duplicate row created.
4. Integration test for inbound sync:
   • DeviceA receives a peer message.
   • Verify DeviceB gets the sync and stores it correctly.