Trellara correctness report
Deterministic failure proof for Postgres CDC
This static report is generated from Trellara's replayable simulator and curated chaos matrix. It is designed for CI publishing so buyers can inspect the failure boundaries instead of trusting a reliability claim.
OverallPASS
Simulations28
Chaos Scenarios63
Seed20260812
CI Contract
Verification command: cargo test --workspace
Mode: deterministic_failure_matrix
Report Identity
Artifact: docs/correctness-report.html
Report version: 2026-08-mvp
Package version: 0.1.0
Source revision: 2583232ab4f8627ad4a5b0a687dc198a0f6652ac
Source repository: trellara/trellara
Workflow run: https://github.com/trellara/trellara/actions/runs/36459738535
Generated by: trellara chaos report
Freshness check: make verify-correctness-report
Performance Envelope
Quickstart target: 8 minute estimate within 10 minute budget
Default local relay/apply bound: 100 transactions / 100 messages
Stream spill threshold: 1024 changes
Stream spill location: quickstart configs set source.stream_spill_dir to ./target/trellara-spill
Large transaction mode: pgoutput protocol v2 streaming with Trellara manifest and commit marker barriers
Local durability acknowledgement: source acknowledgement advances only after local durable append
Local indexed replay: stream inspection rebuilds missing, stale, or corrupt sidecar indexes from durable log frames and exposes last valid offsets plus seek guidance for replay
Enterprise Proof Review Path
This report is the public failure-evidence leg of the same proof chain exported by trellara pilot-package. Reviewers should connect the deterministic failure matrix to live source, boundary, convergence, and recovery artifacts before approving a production pilot.
| Review question | Proof surface | Enterprise gate |
|---|---|---|
| Is the source safe to capture? | trellara check --config <flow> --format text | No critical slot, WAL retention, replica identity, failover-slot, or subscription conflict blockers |
| Does CDC preserve transaction boundaries? | trellara inspect-transaction --file <envelope.pb> --format text | Checksum status is match, affected tables are explicit, manifest boundary_mode names strict chunk or partition semantics, and manifest counts bind to the commit marker |
| Can a first snapshot hand off to the stream safely? | trellara snapshot --config <flow> --run-id <run> | Every selected table reaches stream_handoff_ready with a durable handoff watermark |
| Has the target converged? | trellara verify --config <flow> | Row counts and checksums match at a caught-up target checkpoint |
| Can operators recover without silent skips? | trellara status --config <flow> --view diagnostics --format text | Quarantine, repair-plan, metrics, and replay-ready commands identify the safe next action |
| Have durable-boundary failure harnesses and observability assertions passed? | trellara chaos report --output docs/correctness-report.html | Source promotion, broker quorum loss, target restart, object-store/catalog split, and 24-hour large-transaction soak all pass with named recovery and metric evidence |
| Can partitioned scale expose global visibility safely? | trellara partition-watermarks --config <flow> | Every partition has checkpoint evidence before global current-state visibility advances |
| Can the proof package be shared and audited? | trellara pilot-package --config <flow> | proof-bundle.md and manifest.json bind the review chain to SHA-256 digests |
Replayable Simulation Suite
| Failure point | Status | Seed | Transactions | Skipped duplicates | Invariant evidence |
|---|---|---|---|---|---|
publish_ack_loss | pass | 20260812 | 8/8 applied | 1 | cargo test -p trellara-sim publish_ack_loss |
crash_after_publish_before_source_ack | pass | 20260813 | 8/8 applied | 1 | cargo test -p trellara-sim crash_after_publish_before_source_ack |
source_failover_after_publish_before_ack | pass | 20260814 | 8/8 applied | 1 | cargo test -p trellara-sim source_failover_after_publish_before_ack |
duplicate_delivery | pass | 20260815 | 8/8 applied | 1 | cargo test -p trellara-sim duplicate_delivery |
target_failure_before_commit | pass | 20260816 | 8/8 applied | 0 | cargo test -p trellara-sim target_failure_before_commit |
target_quarantine_repair_replay | pass | 20260817 | 8/8 applied | 0 | cargo test -p trellara-sim target_quarantine_repair_replay |
checkpoint_failure_during_apply | pass | 20260818 | 8/8 applied | 0 | cargo test -p trellara-sim checkpoint_failure_during_apply |
stream_ack_loss_after_apply | pass | 20260819 | 8/8 applied | 1 | cargo test -p trellara-sim stream_ack_loss_after_apply |
Snapshot Handoff Simulation Suite
| Failure point | Status | Seed | Tables | Post-snapshot replay | Invariant evidence |
|---|---|---|---|---|---|
snapshot_source_crash_during_table_copy | pass | 20260912 | 4/4 copied | 6/6 replayed | cargo test -p trellara-sim snapshot_source_crash_during_table_copy |
snapshot_relay_crash_during_table_copy | pass | 20260913 | 4/4 copied | 6/6 replayed | cargo test -p trellara-sim snapshot_relay_crash_during_table_copy |
snapshot_target_crash_during_table_copy | pass | 20260914 | 4/4 copied | 6/6 replayed | cargo test -p trellara-sim snapshot_target_crash_during_table_copy |
snapshot_duplicate_copy_attempt | pass | 20260915 | 4/4 copied | 6/6 replayed | cargo test -p trellara-sim snapshot_duplicate_copy_attempt |
snapshot_ddl_during_table_copy | pass | 20260916 | 4/4 copied | 6/6 replayed | cargo test -p trellara-sim snapshot_ddl_during_table_copy |
snapshot_handoff_recorded_before_stream_start | pass | 20260917 | 4/4 copied | 6/6 replayed | cargo test -p trellara-sim snapshot_handoff_recorded_before_stream_start |
Strict Chunk Simulation Suite
| Failure point | Status | Seed | Chunks | Duplicates | Apply boundary | Invariant evidence |
|---|---|---|---|---|---|---|
strict_chunk_relay_crash_after_chunks_before_manifest | pass | 20261012 | 4/4 published | 4 | 1 applied after manifest | cargo test -p trellara-sim strict_chunk_relay_crash_after_chunks_before_manifest |
strict_chunk_relay_crash_after_manifest_before_source_ack | pass | 20261013 | 4/4 published | 0 | 1 applied after manifest | cargo test -p trellara-sim strict_chunk_relay_crash_after_manifest_before_source_ack |
strict_chunk_manifest_arrives_with_missing_chunk | pass | 20261014 | 4/4 published | 0 | 1 applied after manifest | cargo test -p trellara-sim strict_chunk_manifest_arrives_with_missing_chunk |
strict_chunk_manifest_arrives_before_chunks | pass | 20261015 | 4/4 published | 0 | 1 applied after manifest | cargo test -p trellara-sim strict_chunk_manifest_arrives_before_chunks |
strict_chunk_target_crash_after_staging_before_commit | pass | 20261016 | 4/4 published | 0 | 1 applied after manifest | cargo test -p trellara-sim strict_chunk_target_crash_after_staging_before_commit |
Fleet Fan-In Lake Simulation Suite
| Failure point | Status | Epoch | Sources | Rows | State | Invariant evidence |
|---|---|---|---|---|---|---|
fleet_fanin_offline_stores_publish_with_gaps | pass | epoch-00000000013528f8 | 9/12 complete, 3 missing, 0 quarantined | 9 transactions / 9 changes, 9 replays | complete_with_gaps | cargo test -p trellara-sim fleet_fanin_offline_stores_publish_with_gaps |
fleet_fanin_late_store_recovery_completes_epoch | pass | epoch-00000000013528f9 | 12/12 complete, 0 missing, 0 quarantined | 12 transactions / 12 changes, 12 replays | complete_with_gaps -> complete | cargo test -p trellara-sim fleet_fanin_late_store_recovery_completes_epoch |
fleet_fanin_duplicate_store_transaction_replay | pass | epoch-00000000013528fa | 9/12 complete, 3 missing, 0 quarantined | 9 transactions / 9 changes, 2 replays | complete_with_gaps | cargo test -p trellara-sim fleet_fanin_duplicate_store_transaction_replay |
fleet_fanin_conflicting_duplicate_quarantine | pass | epoch-00000000013528fb | 8/12 complete, 3 missing, 1 quarantined | 9 transactions / 9 changes, 1 replays | quarantined | cargo test -p trellara-sim fleet_fanin_conflicting_duplicate_quarantine |
Lane D Qualification Suite
| Failure point | Status | Durable boundary | Transactions | Soak/load | Observability assertions | Invariant evidence |
|---|---|---|---|---|---|---|
qualification_source_promotion_while_relay_disconnected | pass | source_failover_slot_to_relay_restartpromoted_source_replays_from_last_durable_ack | 12/12 applied, 1 replayed | failure harness | source_failover_promotion_detected: status reports promoted source identity plus failover slot replaysource_ack_lag_visible: metrics expose acknowledgement lag while relay is disconnected | cargo test -p trellara-sim qualification_source_promotion_while_relay_disconnectedtrellara source-safety --config <flow> --format text; trellara relay --config <flow> |
qualification_broker_outage_quorum_loss | pass | broker_quorum_publish_acksource_ack_waits_for_broker_quorum | 12/12 applied, 0 replayed | failure harness | broker_quorum_unavailable: readyz and metrics mark broker quorum unavailable before source feedbackpublish_retry_recovered: publish retry counter increments after quorum is restored | cargo test -p trellara-sim qualification_broker_outage_quorum_losstrellara status --config <flow> --view alerts --format text; trellara relay --config <flow> |
qualification_target_restart_during_apply | pass | target_apply_transaction_committarget_restart_replays_uncheckpointed_apply | 12/12 applied, 1 replayed | failure harness | target_restart_replay_required: diagnostics name the uncheckpointed apply boundary and replay commandtarget_checkpoint_not_advanced_early: target checkpoint stays behind until redelivery commits | cargo test -p trellara-sim qualification_target_restart_during_applytrellara status --config <flow> --view diagnostics --format text; trellara apply --config <flow> |
qualification_object_store_success_catalog_timeout | pass | object_store_write_before_catalog_visibilitycatalog_timeout_cannot_publish_uncommitted_epoch | 12/12 applied, 0 replayed | failure harness | catalog_commit_pending: lake completeness marks the epoch pending_catalog_commitcatalog_retry_idempotent: catalog retry discovers the existing data-file receipt before release | cargo test -p trellara-sim qualification_object_store_success_catalog_timeouttrellara lake fanin verify --config <flow> --format json |
qualification_twenty_four_hour_soak_large_transaction_memory_ceiling | pass | large_transaction_stream_spill_memory_ceilingsoak_large_transactions_stay_within_memory_ceiling | 12/12 applied, 0 replayed | 24h soak, 50000 changes, 64 / 128 MiB | soak_window_completed: qualification records a 24-hour soak window with passing assertionslarge_transaction_memory_ceiling: peak relay memory remains below the configured ceiling | cargo test -p trellara-sim qualification_twenty_four_hour_soak_large_transaction_memory_ceilingtrellara performance --config <flow> --format json; trellara chaos report --output docs/correctness-report.html |
Curated Failure Matrix
| Scenario | Status | Boundary | Invariant | Safety property | Proof | Recovery |
|---|---|---|---|---|---|---|
| relay_before_broker_publish | covered | strict_transaction_order | source_checkpoint_after_broker_ack | source durable checkpoint is not advanced, so the transaction is captured again | cargo test -p trellara-relay failed_publish_does_not_advance_checkpoint | automatic |
| relay_after_broker_publish_before_source_feedback | covered | strict_transaction_order | source_checkpoint_after_broker_ack | duplicate stream delivery is possible, but no committed transaction is missed | cargo test -p trellara-relay ambiguous_publish_replays_duplicate_without_advancing_checkpoint | automatic |
| source_failover_after_publish_before_ack | covered | strict_transaction_order_with_failover_slot | failover_slot_replay_preserves_transaction_boundary | relay restart on the promoted source may redeliver the last transaction, but downstream dedup preserves exactly-once apply | cargo test -p trellara-sim source_failover_after_publish_before_ack_recovers_with_duplicate_replay | trellara check --config <flow> |
| source_failover_slot_unsynced | covered | source_failover_readiness | failover_slot_sync_before_promotion | source-safety warns before operators rely on a promoted standby that may not retain the CDC boundary | cargo test -p trellara-cli --lib source_safety_warns_when_failover_slot_is_not_synced | trellara check --config <flow> --format text |
| source_failover_slot_disabled | covered | source_failover_readiness | failover_slot_enabled_before_promotion | source-safety reports the disabled failover posture as an explicit recovery boundary before CDC depends on promotion | cargo test -p trellara-cli --lib source_safety_warns_when_failover_slot_is_disabled && cargo test -p trellara-cli --lib direct_source_safety_warns_when_failover_slot_is_disabled | trellara check --config <flow> --format text |
| source_slot_abandoned_idle_timeout | covered | source_safety_slot_lifecycle | inactive_slot_cleanup_posture_is_explicit | source-safety degrades the flow, names inactive-since metadata, and distinguishes disabled from configured idle_replication_slot_timeout cleanup | cargo test -p trellara-cli --lib direct_source_safety_mentions_disabled_idle_slot_timeout && cargo test -p trellara-cli --lib direct_source_safety_surfaces_configured_idle_slot_timeout | trellara-check <source> --format text |
| broker_publish_failure | covered | strict_transaction_order | source_checkpoint_after_broker_ack | relay does not advance source checkpoint without broker acknowledgement | cargo test -p trellara-relay failed_publish_does_not_advance_checkpoint | automatic |
| local_stream_index_rebuild | covered | brokerless_local_stream | local_index_rebuild_preserves_replay_offsets | the local stream rebuilds the derived offset index from durable log frames, excludes torn tails, and reports the last valid replay offset plus locate/seek commands | cargo test -p trellara-stream-local missing_index_is_rebuilt_for_offset_replay --lib && cargo test -p trellara-stream-local stale_index_discovers_durable_tail_without_truncating --lib && cargo test -p trellara-stream-local inspect_reports_corrupt_index_rebuild --lib && cargo test -p trellara-stream-local inspect_reports_torn_tail_bytes_before_recovery_append --lib && cargo test -p trellara-cli local_stream_inspect_summary_reports_depth_and_pending_messages --lib | trellara stream inspect-local --config <flow> |
| local_stream_publish_ack_after_fsync_durability | covered | brokerless_local_stream | source_ack_after_local_fsync_durability | a returned publish acknowledgement identifies an offset whose frame, sidecar index entry, and zero-torn-tail inspection state are immediately replayable under default fsync durability | cargo test -p trellara-stream-local fsync_publish_ack_follows_durable_frame_and_index --lib | trellara stream inspect-local --config <flow> |
| applier_before_target_commit | covered | strict_transaction_order | target_checkpoint_same_transaction_as_apply | target checkpoint and dedup state are not advanced | cargo test -p trellara-apply-postgres missing_target_table_fails_closed_without_checkpoint_or_dedup | trellara quarantine list --config <flow> |
| applier_after_target_commit_before_stream_ack | covered | strict_transaction_order | transaction_dedup_before_reapply | transaction-level dedup makes redelivery safe and the replay is acknowledged | cargo test -p trellara-apply-postgres apply_worker_replays_safely_when_ack_fails_after_apply | automatic |
| duplicate_transaction_replay | covered | strict_transaction_order | transaction_dedup_before_reapply | second delivery is skipped without reapplying target changes | cargo test -p trellara-apply-postgres applies_transaction_once_and_records_checkpoint | automatic |
| target_schema_missing | covered | strict_transaction_order | fail_closed_target_contract | transaction is quarantined without advancing checkpoint or dedup state | cargo test -p trellara-apply-postgres missing_target_table_fails_closed_without_checkpoint_or_dedup | trellara quarantine replay-ready --config <flow> --transaction-id <tx> --commit-lsn <lsn> |
| target_update_delete_zero_rows | covered | strict_transaction_order | no_silent_target_divergence | transaction is quarantined as no_rows_matched without advancing checkpoint or dedup state | cargo test -p trellara-apply-postgres update_delete_zero_row_match_fails_closed | trellara verify --config <flow>; trellara reseed --config <flow> |
| filtered_reseed_preserves_out_of_scope_rows | covered | filtered_reseed_repair | row_filter_reseed_scope_is_preserved | reseed deletes and replaces only rows matching the configured row_filter while preserving target-owned columns and out-of-scope target rows | cargo test -p trellara-verify --test postgres_snapshot_integration postgres_reseed_with_row_filter_replaces_only_matching_target_rows | trellara reseed --config <flow> --table <schema.table> |
| pgoutput_schema_change_during_stream | covered | pgoutput_relation_metadata | source_schema_fingerprint_fail_closed | capture stops before assembling changes under an unexpected relation schema | cargo test -p trellara-pg-capture pgoutput_decoder_fails_closed_on_relation_schema_change && cargo test -p trellara-pg-capture pgoutput_decoder_fails_closed_on_schema_change_during_stream | trellara contract-test --config <flow> |
| pgoutput_row_without_relation_metadata | covered | pgoutput_relation_metadata | relation_metadata_required_before_rows | capture fails closed before constructing a transaction envelope with unknown column identity | cargo test -p trellara-pg-capture pgoutput_decoder_fails_closed_without_relation_metadata | trellara contract-test --config <flow> |
| source_schema_handoff_recovery | covered | pgoutput_relation_metadata | schema_drift_requires_fresh_snapshot_handoff | operators receive a scripted schema-discover, contract-test, fresh snapshot handoff, resume, and verify sequence before CDC continues | cargo test -p trellara-cli --lib source_schema_drift_recovery_action_requires_fresh_handoff && cargo test -p trellara-cli --lib contract_test_scripts_schema_handoff_when_pinned_fingerprint_drifts | trellara repair-plan --config <flow> |
| pgoutput_dml_truncate_envelope_mapping | covered | pgoutput_relation_metadata | pgoutput_dml_maps_to_transaction_envelope | each pgoutput operation maps to the expected Trellara envelope operation with relation identity, row images, source ordering, and checksum evidence | cargo test -p trellara-pg-capture pgoutput_decoder_tracks_relation_and_insert_tuple && cargo test -p trellara-pg-capture pgoutput_decoder_parses_update_key_and_omits_unchanged_toast && cargo test -p trellara-pg-capture pgoutput_decoder_parses_delete_truncate_and_commit_boundaries && cargo test -p trellara-pg-capture assembler_expands_truncate_relations_in_source_order | trellara inspect-transaction --file <envelope.pb> |
| pgoutput_ddl_only_boundary | covered | pgoutput_transaction_boundary | ddl_only_transaction_preserves_commit_boundary | capture emits a valid DDL-only transaction envelope with the source transaction id, commit LSN, schema version evidence, and post-DDL DML release gate | cargo test -p trellara-pg-capture assembler_emits_ddl_only_transaction | trellara schema ddl-envelope-plan --config <flow> --file <envelope.pb> |
| pgoutput_mixed_ddl_dml_boundary | covered | pgoutput_transaction_boundary | mixed_ddl_dml_transaction_preserves_total_order | capture keeps DDL and DML in one transaction envelope, preserves source total order, and forces downstream release through the DDL barrier | cargo test -p trellara-pg-capture assembler_preserves_mixed_ddl_and_dml_total_order | trellara inspect-transaction --file <envelope.pb> |
| pgoutput_streamed_transaction_spills_until_commit | covered | pgoutput_protocol_v2_streaming | streamed_transaction_visible_only_after_stream_commit | capture may spill buffered changes to disk, but emits one committed transaction only after STREAM COMMIT | cargo test -p trellara-pg-capture assembler_emits_streamed_transaction_only_on_stream_commit | automatic |
| pgoutput_stream_abort_discards_partial_changes | covered | pgoutput_protocol_v2_streaming | stream_abort_discards_partial_changes | aborted stream fragments are removed from memory and spill files, and no partial transaction becomes visible | cargo test -p trellara-pg-capture assembler_drops_aborted_stream_subtransaction_changes && cargo test -p trellara-pg-capture assembler_removes_spill_file_after_stream_abort | trellara relay --config <flow> |
| pgoutput_keepalive_reports_last_durable_ack | covered | pgoutput_replication_protocol | keepalive_ack_uses_last_durable_boundary | the standby status update repeats only the last durable acknowledged LSN for written, flushed, and applied watermarks | cargo test -p trellara-pg-capture standby_status_boundary_uses_only_last_durable_acknowledged_lsn && cargo test -p trellara-pg-capture standby_status_update_payload_reports_acknowledged_lsn_for_every_watermark | trellara status --config <flow> --view report --format text |
| missing_replica_identity | covered | strict_transaction_order | preflight_before_cdc_start | preflight and contract-test fail before relay starts | cargo test -p trellara-pg-capture capture_preflight_reports_replica_identity_safety | trellara contract-test --config <flow> |
| default_replica_identity_primary_key_apply | covered | strict_transaction_order | primary_key_predicate_apply_without_full | UPDATE and DELETE statements use key predicates, so ordinary primary-key tables do not require REPLICA IDENTITY FULL | cargo test -p trellara-apply-postgres plans_update_with_key_predicate | trellara contract-test --config <flow> |
| default_replica_identity_key_change_apply | covered | strict_transaction_order | primary_key_moves_match_old_key_and_set_new_key | UPDATE statements match the old key from the before image and assign the new key from the after image in the same target transaction | cargo test -p trellara-apply-postgres key_changing_update_sets_new_key_and_matches_old_key && cargo test -p trellara-apply-postgres --test postgres_integration key_changing_update_moves_primary_key_with_old_key_predicate | trellara contract-test --config <flow> |
| unchanged_toast_columns_preserved | covered | pgoutput_relation_metadata | absent_toast_columns_are_unchanged | apply plans omit absent or explicit unchanged TOAST markers for non-key columns so target data is preserved | cargo test -p trellara-apply-postgres update_omits_absent_non_key_columns_for_unchanged_toast && cargo test -p trellara-apply-postgres update_omits_explicit_unchanged_toast_marker | trellara contract-test --config <flow> |
| snapshot_duplicate_copy_attempt | covered | initial_snapshot_to_stream_handoff | snapshot_copy_idempotent_by_run_and_table | completed table copies are reported as skipped instead of creating a second initial copy | cargo test -p trellara-sim snapshot_duplicate_copy_attempt_skips_completed_table_before_handoff && cargo test -p trellara-cli completed_snapshot_summary_reuses_handoff_ready_run | automatic |
| snapshot_incomplete_copy_resume | covered | initial_snapshot_to_stream_handoff | snapshot_handoff_requires_complete_tables | active or incomplete table progress is not falsely treated as handoff-ready | cargo test -p trellara-cli completed_snapshot_summary_requires_terminal_run_and_complete_tables | trellara snapshot --config <flow> --run-id <run> |
| snapshot_source_crash_during_copy | covered | initial_snapshot_to_stream_handoff | snapshot_source_failure_retries_before_handoff | source copy failure marks the run recoverable, retries the table, and records handoff only after all selected tables are complete | cargo test -p trellara-sim snapshot_source_crash_during_table_copy_retries_before_handoff | trellara snapshot --config <flow> --run-id <run> |
| snapshot_relay_crash_during_copy | covered | initial_snapshot_to_stream_handoff | snapshot_copy_idempotent_by_table | the table copy is retried and handoff is recorded only after every selected table is complete | cargo test -p trellara-sim snapshot_relay_crash_during_table_copy_retries_before_handoff | trellara snapshot --config <flow> --run-id <run> |
| snapshot_target_crash_during_copy | covered | initial_snapshot_to_stream_handoff | snapshot_copy_no_handoff_until_complete | target copy is retried and stream handoff is withheld until convergence proof can include all tables | cargo test -p trellara-sim snapshot_target_crash_during_table_copy_retries_before_handoff | trellara snapshot --config <flow> --run-id <run> |
| snapshot_copy_failure_marked_recoverable | covered | initial_snapshot_to_stream_handoff | snapshot_failure_state_is_durable_before_retry | snapshot run and table progress are marked failed_recoverable before retry, and correctness reports surface the recovery action | cargo test -p trellara-sim snapshot_relay_crash_during_table_copy_retries_before_handoff && cargo test -p trellara-cli --lib snapshot_copy_failure_records_recoverable_run_and_table_state && cargo test -p trellara-cli --lib correctness_report_surfaces_recoverable_snapshot_copy_failure | trellara snapshot --config <flow> --run-id <run> |
| snapshot_invalid_state_regression | covered | initial_snapshot_to_stream_handoff | snapshot_state_machine_monotonic_until_recoverable | unsafe backward transitions are rejected unless the run is explicitly marked recoverable | cargo test -p trellara-checkpoint snapshot_run_state_rejects_unsafe_backward_transitions | trellara snapshot --config <flow> --run-id <run> --force |
| snapshot_exported_visibility_boundary | covered | initial_snapshot_to_stream_handoff | snapshot_copy_uses_exported_lsn_boundary | initial copy sees only rows visible in the exported snapshot; later writes must arrive through the stream | cargo test -p trellara-verify postgres_reseed_imports_exported_source_snapshot | automatic |
| partition_manifest_missing_chunk | covered | partitioned_scale_mode | manifest_barrier_before_partition_apply | barrier reconstruction rejects incomplete, duplicate, or malformed manifest partition boundaries before target apply | cargo test -p trellara-protocol barrier_reconstruction_rejects_missing_chunks && cargo test -p trellara-protocol barrier_reconstruction_rejects_exact_duplicate_chunks && cargo test -p trellara-protocol barrier_reconstruction_rejects_duplicate_manifest_partitions && cargo test -p trellara-protocol commit_marker_rejects_duplicate_manifest_partitions | trellara partition-watermarks --config <flow> |
| partition_chunk_routing_header_mismatch | covered | partitioned_scale_mode | partition_chunk_headers_are_apply_trust_boundary | stream producers and target appliers reject mismatched partition id, event count, checksum, transaction identity, or unlisted manifest membership before buffering, applying, or acknowledging the chunk, and runtime pending stats expose extra chunk blockers | cargo test -p trellara-stream chunk_message_rejects_envelope_transaction_mismatch --lib && cargo test -p trellara-apply-postgres --lib barrier_worker_rejects_chunk_partition_routing_header_mismatch && cargo test -p trellara-apply-postgres --lib barrier_worker_rejects_unlisted_chunk_buffered_before_manifest && cargo test -p trellara-apply-postgres --lib barrier_worker_rejects_unlisted_chunk_after_manifest && cargo test -p trellara-apply-postgres pending_stats_reports_extra_chunks_separately && cargo test -p trellara-stream-local --lib reconstruct_local_barrier_transaction_rejects_chunk_header_payload_mismatch | trellara run --local --verify --config <flow> |
| partition_commit_marker_ack_ambiguous | covered | partitioned_scale_mode | partition_source_checkpoint_waits_for_commit_marker_ack | source durable checkpoint and source feedback do not advance until the partition chunk, manifest, and commit marker are all acknowledged | cargo test -p trellara-relay partitioned_ambiguous_commit_marker_publish_does_not_advance_checkpoint | trellara relay --config <flow> |
| partition_commit_marker_manifest_mismatch | covered | partitioned_scale_mode | partition_commit_marker_matches_manifest_before_apply | stream producers reject mismatched barrier payload identity; the barrier-aware applier rejects mismatched commit markers before target apply or acknowledgement, and runtime pending stats expose invalid commit-marker blockers | cargo test -p trellara-stream manifest_message_rejects_envelope_boundary_mismatch --lib && cargo test -p trellara-stream commit_marker_message_rejects_envelope_boundary_mismatch --lib && cargo test -p trellara-apply-postgres barrier_worker_rejects_mismatched_commit_marker && cargo test -p trellara-apply-postgres pending_stats_reports_invalid_commit_marker_separately && cargo test -p trellara-cli apply_summary_exposes_invalid_commit_marker_blocker | trellara status --config <flow> --view report --format text |
| strict_chunk_manifest_missing_chunk | covered | strict_chunked_transaction_order | strict_chunk_manifest_barrier_before_apply | barrier reconstruction rejects incomplete large transactions before target apply | cargo test -p trellara-sim strict_chunk_missing_chunk_waits_for_replay_before_apply | trellara status --config <flow> --view report --format text |
| strict_chunk_out_of_order_arrival | covered | strict_chunked_transaction_order | strict_chunk_manifest_waits_for_completeness | barrier-aware apply buffers the manifest and waits for every chunk before applying | cargo test -p trellara-sim strict_chunk_manifest_before_chunks_waits_for_complete_chunk_set | trellara apply --config <flow> |
| strict_chunk_relay_crash_after_chunks_before_manifest | covered | strict_chunked_transaction_order | strict_chunk_manifest_required_before_apply | target waits for the manifest and commit marker barrier, replayed chunks are deduplicated, and no partial transaction is applied | cargo test -p trellara-sim strict_chunk_crash_after_chunks_waits_for_manifest_before_apply | trellara relay --config <flow> |
| strict_chunk_partial_publish_failure | covered | strict_chunked_transaction_order | source_checkpoint_waits_for_all_barrier_messages | source durable checkpoint and source feedback do not advance until every chunk, manifest, and commit marker publish succeeds | cargo test -p trellara-relay strict_chunked_partial_publish_failure_does_not_advance_checkpoint_or_source_ack | trellara relay --config <flow> |
| strict_chunk_relay_crash_after_manifest_before_source_ack | covered | strict_chunked_transaction_order | strict_chunk_source_ack_after_manifest_publish | source checkpoint remains behind until the complete chunk set and manifest can be replayed safely | cargo test -p trellara-sim strict_chunk_crash_after_manifest_recovers_without_moving_source_ack_early | trellara relay --config <flow> |
| strict_chunk_target_crash_after_staging_before_commit | covered | strict_chunked_transaction_order | strict_chunk_stage_not_visible_before_target_commit | uncommitted staged chunks are discarded on restart, the manifest is replayed, and the target applies the transaction exactly once | cargo test -p trellara-sim strict_chunk_target_crash_after_staging_applies_once_after_replay | trellara apply --config <flow> |
| protocol_property_chunk_manifest_reconstruction | covered | strict_chunked_or_partitioned_barrier | manifest_barriers_reconstruct_source_order_for_all_generated_inputs | property tests reconstruct every generated strict chunk or partitioned transaction in source order and reject missing chunks with minimized failing repros | cargo test -p trellara-protocol strict_chunk_manifest_property_reconstructs_source_order && cargo test -p trellara-protocol partitioned_manifest_property_reconstructs_source_order | automatic |
| strict_chunk_checksum_tampering | covered | strict_chunked_transaction_order | chunk_checksum_mismatch_fails_closed | barrier reconstruction rejects the transaction before apply when a chunk checksum no longer matches the manifest | cargo test -p trellara-protocol barrier_reconstruction_rejects_chunk_checksum_tampering | trellara status --config <flow> --view report --format text |
| barrier_conflicting_duplicate_messages | covered | strict_chunked_or_partitioned_barrier | conflicting_duplicate_barrier_messages_fail_closed | the barrier-aware applier rejects conflicting duplicate chunks, manifests, and commit markers before target apply or stream acknowledgement | cargo test -p trellara-apply-postgres barrier_worker_rejects_conflicting_duplicate_chunks && cargo test -p trellara-apply-postgres barrier_worker_rejects_conflicting_duplicate_manifests && cargo test -p trellara-apply-postgres barrier_worker_rejects_conflicting_duplicate_commit_markers_before_manifest | trellara status --config <flow> --view report --format text |
| partition_applier_before_ack | covered | partitioned_scale_mode | manifest_checkpoint_same_transaction_as_apply | partition checkpoints and dedup state make redelivery safe without partial visibility | cargo test -p trellara-apply-postgres manifest_envelope_records_partition_checkpoints_atomically | trellara status --config <flow> --view dashboard --format text |
| fleet_fanin_offline_stores_publish_with_gaps | covered | fleet_fanin_epoch_completeness | lake_epoch_gaps_are_explicit | a publish-with-gaps policy marks missing sources explicitly and only exposes the epoch as complete_with_gaps | cargo test -p trellara-sim fleet_fanin_offline_stores_publish_with_explicit_gap_state | trellara lake plan --config <flow> --format json |
| fleet_fanin_late_store_recovery | covered | fleet_fanin_epoch_completeness | late_sources_recompute_epoch_completeness | late source envelopes can recompute the same epoch from complete_with_gaps to complete without double counting prior stores | cargo test -p trellara-sim fleet_fanin_late_sources_recompute_epoch_to_complete | trellara lake plan --config <flow> --format json |
| fleet_fanin_duplicate_store_replay | covered | fleet_fanin_epoch_completeness | fleet_fanin_deduplicates_by_transaction_boundary | identical source transaction replays are skipped so epoch transaction and change counts stay stable | cargo test -p trellara-sim fleet_fanin_duplicate_store_replay_is_deduplicated | automatic |
| fleet_fanin_conflicting_duplicate_quarantine | covered | fleet_fanin_epoch_completeness | fleet_fanin_conflicting_duplicates_fail_closed | conflicting duplicate evidence quarantines the epoch instead of publishing an ambiguous current-state or SCD2 view | cargo test -p trellara-sim fleet_fanin_conflicting_duplicate_quarantines_epoch | trellara status --config <flow> --view diagnostics --format text |
| qualification_source_promotion_while_relay_disconnected | covered | source_failover_slot_to_relay_restart | promoted_source_replays_from_last_durable_ack | relay restart on the promoted source replays from the last durable acknowledgement, downstream dedup absorbs the replay, and source acknowledgement lag remains visible while disconnected | cargo test -p trellara-sim qualification_source_promotion_while_relay_disconnected | trellara check --config <flow> --format text; trellara relay --config <flow> |
| qualification_broker_outage_quorum_loss | covered | broker_quorum_publish_ack | source_ack_waits_for_broker_quorum | source feedback is withheld during quorum loss, publish retry is observable, and acknowledgement advances only after broker quorum returns | cargo test -p trellara-sim qualification_broker_outage_quorum_loss | trellara status --config <flow> --view alerts --format text; trellara relay --config <flow> |
| qualification_target_restart_during_apply | covered | target_apply_transaction_commit | target_restart_replays_uncheckpointed_apply | uncheckpointed apply state rolls back, redelivery applies the transaction exactly once, and diagnostics name the replay boundary | cargo test -p trellara-sim qualification_target_restart_during_apply | trellara status --config <flow> --view diagnostics --format text; trellara apply --config <flow> |
| qualification_object_store_success_catalog_timeout | covered | object_store_write_before_catalog_visibility | catalog_timeout_cannot_publish_uncommitted_epoch | the epoch remains pending_catalog_commit, catalog retry discovers the existing object-store receipt, and downstream Spark consumption is held until catalog evidence is durable | cargo test -p trellara-sim qualification_object_store_success_catalog_timeout | trellara lake fanin verify --config <flow> --format json |
| qualification_twenty_four_hour_soak_large_transaction_memory_ceiling | covered | large_transaction_stream_spill_memory_ceiling | soak_large_transactions_stay_within_memory_ceiling | the qualification harness records a 24-hour soak window, proves large transactions spill before publish, and asserts peak relay memory stays below the configured ceiling | cargo test -p trellara-sim qualification_twenty_four_hour_soak_large_transaction_memory_ceiling | trellara performance --config <flow> --format json; trellara chaos report --output docs/correctness-report.html |
| snapshot_ddl_during_copy | covered | initial_snapshot_to_stream_handoff | snapshot_contract_before_handoff | schema drift withholds handoff until contract refresh and table-copy retry complete | cargo test -p trellara-sim snapshot_ddl_during_table_copy_withholds_handoff_until_contract_refresh | trellara snapshot --config <flow> --run-id <run> --force |
| snapshot_handoff_recorded_before_stream_start | covered | initial_snapshot_to_stream_handoff | snapshot_handoff_before_stream_checkpoint | post-snapshot writes are replayed only after the stream restarts from the recorded handoff boundary | cargo test -p trellara-sim handoff_recorded_before_stream_start_recovers_before_replay | trellara relay --config <flow> |
| repair_and_replay_ready | covered | strict_transaction_order_or_partitioned_scale_mode | operator_replay_preserves_transaction_boundary | dedup/quarantine state can be cleared so redelivery applies instead of skipping | cargo test -p trellara-sim target_quarantine_repair_replay_applies_after_operator_marks_replay_ready | trellara quarantine replay-ready --config <flow> --transaction-id <tx> --commit-lsn <lsn> |
| quarantine_replay_ready_requires_exact_boundary | covered | strict_transaction_order_or_partitioned_scale_mode | replay_ready_requires_exact_quarantine_boundary | replay-ready fails closed unless the exact quarantined transaction boundary exists, so operators do not clear dedup or request redelivery for the wrong CDC transaction | cargo test -p trellara-cli quarantine_replay_ready_refuses_unknown_transaction_boundary | trellara quarantine list --config <flow> |