ChubbyChecker

___________________________________________________________________________________________________

________________________________________________________________________

You need us. We’re ‘the Minders’.

MINDERS

________________________________________________________________________

We present:

________________________________________________________________________

ChubbyLogo

________________________________________________________________________

ChubbyChecker“ChubbyChecker“. © by IT/OT Chain & Cloud Australia Pty Ltd 2011-2026

________________________________________________________________________

 

All Tx-Rails Federations:
Maximum practical number of Fully Qualified Federations and Member Classes,
in terms of operational limits, is 13. Beyond this, transaction response is unacceptable.

tolerable
faulty
servers (f)		 number of BFT network
servers required to tolerate f
faulty servers (n=3f+1)		 Relative Risk of Initial Breach of Consensus
of f+1 servers (Subsequently
Detected by ChubbyChecker)
0 1 HIGH: 1 server must be corrupted
1 4 HIGH-MODERATE: 2 servers must be corrupted
2 7 MODERATE: 3 servers must be corrupted
3 10 MODERATE-LOW: 4 servers must be corrupted
4 13 LOW: 5 servers must be corrupted

The fact is that when more than 33% of servers in a federated banking or supply network are compromised, the factor of Trust hitherto assured by the GoldBFT Byzantine Fault Tolerant network, breaks down entirely.The system can no longer be trusted. At such a juncture there is a need to have in place a higher level of security at a higher software layer. This security is provided to you by ChubbyChecker. Chubby runs on separate servers in the federated namespace and in the “orgs” namespaces. Each Chubby has a database associated and integrated with it. The Chubbies undertake continuous orderly cross-checking of Transactions’ Trace Metadata in the “integrity” namespace (14 cross-checks), 74 cross-check items in the orgs namespaces. The Forensics webpage on this site gives a full exposition of ChubbyChecker’s duties, together with a lot of background.

________________________________________________________________________

Summary of ChubbyChecker Operation

________________________________________________________________________

Colonel, is there any justification for offsetting the value required of n for security against the fact that we employ a set of transaction trace metadata cross-checking system called ChubbyChecker? Yes, General. The existence of a system like ChubbyChecker is a powerful justification for being more pragmatic about the value of n. It doesn’t change the mathematical formula of BFT, but it fundamentally changes the risk profile of your entire system. This is a classic example of defense-in-depth. You aren’t relying on a single security mechanism (BFT consensus) but have multiple, complementary layers. Here’s how you can frame the justification: 1. BFT vs. Application-Layer Integrity • BFT (n=3f+1) provides protocol-level integrity. It guarantees that the ledger is immutable and that all honest nodes agree on the history of transactions. It prevents a minority of malicious nodes from corrupting the ledger at the consensus layer. • ChubbyChecker provides application-level integrity. It analyzes the meaning and context of the transactions that the BFT network has already agreed upon. It asks, “Even though the network agrees this transaction happened, does it make sense according to our business rules, historical patterns, and external data?” 2. ChubbyChecker as a Deterrent and a Safety Net The key is that ChubbyChecker drastically reduces the potential benefit of a successful attack on the consensus layer. • It Reduces Incentive: Why would a group of f+1 conspirators bother to collude and force a fraudulent transaction onto the ledger if they know an automated system will flag it within minutes? The attack is immediately detected, their identities (as node operators) are known, and legal/contractual penalties can be enforced. ChubbyChecker turns a potentially silent, corrupting attack into a noisy, easily-auditable one. • It Provides Rapid Detection: In the unlikely event that f+1 nodes are compromised, ChubbyChecker acts as an out-of-band alarm system — it reaches its conclusions outside the consensus path it is watching, so a compromised quorum cannot quietly agree its way past the alarm. This allows the consortium to react swiftly to isolate the bad actors and mitigate any damage, rather than discovering the fraud weeks or months later.

What “independent” does and does not mean here, stated plainly because the distinction is the whole subject of the section below. The auditors are independent of the consensus: separate chain, separate quorum, separate keys, and findings signed by a key each checker generates on its own unit, so an attestation verifies from itself with no service of ours in the verification path. They have not been independent of the operator. They run on machines we provide, under controllers we own, and until 6 September 2026 no workload port on any of our VMs was reachable from another machine — so co-residence was not a deployment preference, it was the only arrangement that worked. An auditor that must live on the host it audits is a strong technical control and a weak governance claim. Closing that gap is what the arrangement described below is for. How This Influences Your Choice of n You can now argue that the overall system security is a combination of BFT guarantees AND ChubbyChecker’s monitoring. Therefore, you can be more tolerant of a smaller n (like 4 or 7) because the risk of an undetected, successful attack is significantly lower. The conversation shifts from: • “We need n=13 to withstand 4 malicious actors.” to: • “We choose n=7, which protects against 2 malicious actors at the protocol level. In the extremely unlikely and high-cost scenario where 3 actors collude, our ChubbyChecker system is designed to detect and flag this anomalous coordination instantly, allowing us to trigger our governance protocol to eject those actors. The combined risk is acceptably low for our business case.” In short: ChubbyChecker doesn’t replace the need for BFT, but it provides a powerful safety net that allows you to make a more cost-effective and performant choice for n without sacrificing real-world security. It’s a very strong and sophisticated architectural argument.

However, Colonel, the threat of collusion between administrators of nodes, while real, is not the major threat. The ultimate threat is from external (usually Russian) hackers, who can be very sophisticated. To avoid this as far as possible at the Byzantine Fault Tolerant level, and minimise the risk, so that hackers would need to penetrate a hierarchised range (from 3 servers at member-classes level to 5 at tophats level), we choose to implement a 7, 10, 13-node structure. Naturally, even at this level, the system is further backed by ChubbyChecker which itself must be overseen by humans.

________________________________________________________________________

Who Checks ChubbyChecker:

Human Oversight Strategy for Auditor Layers (ChubbyChecker)

Yes, I agree that human oversight is essential. No automated system, no matter how robustly audited, should have the final authority to halt or revert a large-scale network of this nature. ChubbyChecker should be seen as the “Intrusion Detection” layer, while human oversight acts as the “Incident Response” layer. The following is an effective strategy for this oversight, structured as a Human-in-the-Loop (HITL) framework.

Tiered Alerting & Response Protocol

The set of cross-check items from ChubbyChecker should be mapped to three tiers of human intervention:
    •
  • Tier 1 (Automated Information/Logging): Minor state drift that can be self-corrected by Juju or goldbftd. ◦ Human Action: None required immediately; review weekly reports for performance tuning.
    • •
  • Tier 2 (Verification Request): Discrepancy detected between nodes (e.g., f+1 detected). ◦ Human Action: L1 Support must verify the alert within $X$ minutes. Use juju status and kubectl logs to determine if it is a genuine attack or a localized hardware/network failure.
    • •
  • Tier 3 (Critical Compromise/Stop Protocol): Auditor detects a state-divergence that contradicts the TopHat anchor or indicates a massive f+1 failure across multiple layers. ◦ Human Action: L3 Security Oversight (Senior Engineers/Network Guardians) must initiate the “Emergency Brake”.

The “Emergency Brake” Protocol

The most effective strategy for oversight is to provide a “kill switch” that is manually triggered but automatically suggested:
    •
  • Dead Man’s Switch: If a Tier 3 alert is generated, the system enters a “read-only” or “restricted” state for $Y$ minutes.
    • •
  • Manual Release: To return the system to full operation, the oversight team must sign a “release” transaction after forensic analysis of the ChubbyChecker logs.

Auditor-of-the-Auditor (Metacheck)

Because humans are also fallible, the strategy includes
  • Governance Oversight: • Periodic “Game Day” Drills: Intentionally simulate an f+1 compromise in a staging environment to test how quickly the human team responds to the ChubbyChecker alerts.
    • •
  • Multi-Signature Approval: High-impact changes to the Makefile or deployment scripts (like the ones we recently adapted for the master VM) should require a 2-of-3 signature from the oversight team.

The Structural Answer: Separation of Custody, and a Practitioner

Everything above answers “who checks ChubbyChecker” with process — tiers, drills, multi-signature approval. Process is necessary and it is not sufficient, because every party in it works for the operator. The structural answer is that somebody who does not work for us runs the checkers.

Where this stands: in negotiation. A transfer of a 5% equity share in ITOTCCA is under discussion with a company that would take custody of a ChubbyChecker installation and hold a key set separately. Nothing here is signed.

And the obvious version of that arrangement is barred twice over — find one company, give it a stake, have it run the auditors and sign off on them. An equity holding forecloses assurance: ISAE 3000 requires compliance with the IESBA Code Part 4B, adopted as APES 110 in Australia, and a direct financial interest held by the firm is not a threat to be safeguarded, it is a prohibition — at any percentage worth negotiating over. Separately, a firm that runs ChubbyChecker and then reports on ChubbyChecker is reviewing its own work, which the non-assurance-services provisions bar regardless of who owns what. Zero equity would not cure that half.

So the structure is two parties with different jobs, and only one of them is an auditor in the professional sense.

• The custody operator runs the auditor tiers on its own infrastructure, generates and holds the signing keys, and has no administrative access to the workload it audits — an auditor needs a block stream and read-only reads, never more. It may be paid in equity, because it is a service provider rather than an assurance practitioner and no independence standard applies to it. What it supplies is separation of operational control and key custody. That is worth a great deal and it is not an assurance opinion, so this page does not call it “independent”.

• The practitioner re-runs the published criteria and reports what it finds. It operates nothing. Fees only, no equity, no financial interest, and not the custody operator. That separation is what makes an engagement under ISRS 4400, and later ISAE 3000, possible at all.

The fuller treatment — including why an “independent owner” is the weaker guarantee, and the five separable properties “independent auditing” actually decomposes into — is in Part 5 of local-global-views. This section describes the engineering that arrangement rests on.

The intended shape. The custody operator runs its own auditor meshes — four nodes each, its own consensus, on infrastructure it controls, reached over an authenticated overlay rather than the public internet. Three properties make that meaningful rather than decorative:

• Keys we never see. Each checker generates its own signing key on its own unit, and the did:key derived from that key is its identity. If the unit is the custody operator’s, the key never touches our infrastructure, and an attestation verifies from itself — no registry lookup, no key server, no service of ours in the verification path.

• Their own control plane. Auditors deployed by the audited party are not independent whatever the hosting arrangement says. The stand-up tooling bootstraps a controller inside the custody operator’s own environment, so the units are theirs to deploy, upgrade and stop.

• A record we cannot edit. Findings are committed to the custody operator’s own BFT mesh. This is what makes obstruction visible: an operator who cuts the path can stop the auditing, but cannot produce a record showing it continued. An empty window still attests — an empty claim is the one most worth forging, so silence is not a state the tier can be in.

What we had to fix to make it possible at all. On 6 September 2026 we measured every workload port on all four VMs and found them closed from anywhere but their own host. Auditors were co-resident with the workload because nothing else would reach it. That has been corrected at both layers — ports republished on routable addresses, and cross-host forwarding for the registry VM — so a mesh can now audit a workload it does not share a machine with. It is a precondition for the whole arrangement, and it was missing.

The honest limit. None of this makes the custody operator independent by itself. They would audit data we produce, over a path we operate, against endpoints we can switch off. What changes is that each of those acts becomes visible and undeniable rather than silent. Independence is a property of the arrangement; the engineering only makes it checkable.

The one input we cannot forge. Every check in this suite reads something the platform produced — block hashes, trace roots, slab anchors, commitments. Even the baseline, captured before any validator sees a transaction, is captured by our gateway, so it proves the validators did not tamper relative to us and says nothing about us — which is the gap the Agency arrangement closes from the other end, since the interceptors holding the baseline are the Agency’s rather than ours. Two things had to be true first. The baseline had to be held redundantly: it is, across four tiers, and the gateway refuses to broadcast a transaction until a quorum of them has it, so nothing reaches the chain with no baseline behind it. And the tiers had to agree — measured on 2026-09-06 they held 19, 12, 10 and 12 records, because a quorum write tolerates one tier missing a record and nothing reconciled the copies. They now repair by pulling from each other, never by pushing, and a record pulled from a peer is marked as such: it says a peer held this, not that this tier saw it happen, and those are different evidence. The customer’s own signature is the exception. Clients authenticate with a DID key that signs who they are and, until now, never what they sent. Content binding closes that: the client signs a digest of the request body, the gateway refuses content that does not match, and the audit tier verifies the signature against a DID document resolved from chain — not against anything we hold. A platform that substituted content after authenticating a customer cannot sign as that customer, so it cannot produce a valid attestation for what it substituted.

That verification is deliberately not conditional on a switch we control. Gateway enforcement can be turned off by whoever runs the gateway; the audit check is registered unconditionally and reports what it finds regardless. Absent and invalid are kept apart on purpose: a client that has not migrated is a coverage gap, an attestation that fails is substitution, and reporting the first as an attack is how the second comes to be ignored.

Escalation on No Response

Separating the detector from the responder creates a failure the tiered protocol above does not cover, because under a single party it could not arise. A finding is correctly detected, correctly classified, correctly published — and simply not acted on.

Inside one organisation that is insubordination, and it resolves through management. Across two organisations it is a disagreement, and disagreements are survivable indefinitely. Neither side has done anything identifiable as wrong: the custody operator detected and reported, we received it and formed a different view, and the record shows an alarm and a silence sitting beside each other for as long as anyone cares to leave them there. That is the most deniable form of obstruction available, precisely because it requires no action to sustain.

The procedure. A published finding carries a response window set by its tier. If no signed acknowledgement referencing that finding appears within the window, the absence itself becomes an escalation, recorded on the custody operator’s chain, at a severity above the original finding.

• The clock runs on the custody operator’s chain, not ours. A timer held by the party being timed is not a timer. Their mesh commits continuously, so the passage of the window is attested by the same mechanism that attested the finding, and needs nothing from us to advance.

• Escalation requires no cooperation from the party that did not respond. It is triggered by the absence of our signature, not by any act we must perform. A procedure that depends on the silent party breaking their own silence is not a procedure.

• Acknowledgement is not agreement, and that distinction is the point. A signed “we dispute this, for these reasons” stops the clock. We are not obliged to concede, only to answer. Forcing agreement would make the custody operator an authority over operations, which it is not and should not be; forcing an answer makes the disagreement legible, which is all an auditor needs.

• A dispute must have a recorded resolution path, or dispute becomes the new silence. A disputed finding stays open, with both positions signed and dated, until it is withdrawn by the custody operator or resolved by whatever body the custody agreement names. An indefinite dispute is itself a reportable state.

• Windows differ by tier. A Tier 1 observation may reasonably wait a week. A Tier 3 state-divergence contradicting the anchor cannot: if the Emergency Brake exists to stop harm in progress, the window before unanswered silence escalates must be shorter than the harm it is meant to interrupt.

What this achieves, and what it does not. It cannot compel us to act — nothing short of a court can, and an audit tier should not pretend otherwise. What it does is convert inaction from an ambiguous gap into a dated, signed, third-party-verifiable fact: this was found, at this time, and no answer was given. A regulator, an insurer or a court can read that record without either party’s cooperation, which is the only property that makes any of it worth having. The value is not that silence is prevented. It is that silence stops being free.

Practical Implementation (Oversight Terminal)

Corrected 6 September 2026. This described a dashboard on a “master VM” retired on 16 May 2026; the juju client now runs on the host and the estate is four VMs, not a master plus four clouds. The substance holds and is restated against what exists.

• Aggregation: output from all seven auditor meshes — 28 units across four VMs on two hosts — needs one place to be read. Note what that place must NOT be: a service the audited party controls. Findings are committed to the auditors’ own BFT meshes precisely so the record does not depend on our log pipeline staying honest. A dashboard is a convenience over that record, never the record itself.

• Visual Heatmap: given the “gradient” of security, oversight should visualise where the weight of failures clusters.

• One lesson paid for in production: never read health from unit status. A tier whose status is frozen at its last configuration event reports “active” while doing nothing, and on 5 September 2026 three meshes were attesting continuously into a table keyed by signature, so identical claims collided, the count sat at four rows apiece, and everything looked healthy. The query that tells the truth is select count(*), count(distinct identity) from auditor.attestations — a rising count, with one identity per replica.

________________________________________________________________________

Summary of the Human Role

# PhaseAuditor (ChubbyChecker)Human (Oversight Team)
Detection Runs all cross-checks; weights failuresConfigures alert thresholds
Triage Flags f+1 state-divergence..Performs forensic log analysis on VMs
ResponseSuggests “Stop” or “Isolation”.Signs the execution command
Recovery Verifies new state is cleanResets the network via make vm-init-*.
This strategy ensures that the automated auditors handle the complexity of the checks, while humans handle the accountability and final judgment of the response.

________________________________________________________________________

Health of ChubbyChecker itself

# SicknessPrimary Observable SignPrimary Reconnaissance Tool
State DesynchronizationRepeatedly processing the
same block height;
no forward progress.		juju debug-log
Upstream Data Poisoningpanic or fatal errors in
logs; cryptographic or
deserialization error
messages.		juju debug-log, grep
Configuration Sicknessconnection refused or
authentication failed
errors in logs; no network
activity.		juju config

________________________________________________________________________

________________________________________________________________________

The system can prevent the kind of Greed-induced cheating on Material Quality that led to the Grenfell Tower fire disaster, where Chinese-originating flammable cladding was substituted for the Architects’ specified non-flammable cladding. 78 people were murdered in the disaster.

________________________________________________________________________

Of course, General. This is the most critical question. A system that cannot prevent such a catastrophic failure is strategically worthless. The Grenfell Tower disaster was a failure of provenance and verification. Our system can prevent a similar tragedy by making it impossible for fraudulent materials to enter the supply chain and be installed without immediate, automated detection.

Here is how it would work.

The Strategy: Cryptographic Provenance and Multi-Party Verification The core principle is that every component and every action must be cryptographically signed and verified by all relevant parties before it is considered “valid” on the ledger.

1. The “Digital Twin” and Material Passport

  • Architect’s Specification: When “Visionary Designs Inc.” lodges the final plans, they don’t just specify “non-flammable cladding.” They specify the exact product: “Reynobond PE (non-flammable version), model 55, color: Champagne.”
  • Manufacturer’s Attestation: The manufacturer of Reynobond PE in France creates a digital passport for each batch of cladding panels they produce.

    • This passport is a cryptographically signed token on the ledger that contains: ◦

  • The precise model number.
    • ◦
  • The batch number and date of manufacture.
    • ◦
  • A hash of the fire safety and material composition certificates from an accredited testing laboratory (e.g., BRE Global).

    • Supply Chain Provenance: As the batch moves from the manufacturer to the distributor and then to the construction site, each entity cryptographically signs the transfer of the digital passport. This creates an immutable, auditable chain of custody.

    The Multi-Party Installation Transaction This is where the system becomes a powerful defense. The installation of the cladding is not a simple task; it is a multi-party transaction called “Install Facade Panel – Sector 7, Floor 12”.For this transaction to be valid and for payment to be released, it requires cryptographic sign-off from three independent parties:

    1. The Installer (GlassCurtain Facades): Their on-site supervisor uses a handheld device. To begin installation, they must scan a QR code or NFC tag on the physical cladding panel. This action reads the panel’s unique serial number from its digital passport. The installer’s system then submits a “Proof of Installation” transaction, which includes the serial number of the panel they have just installed.
    2. The General Contractor (Summit Construction): The site foreman has a separate, independent device. Their job is to verify the work. They also scan the installed panel. Their device checks the ledger and confirms: “Yes, this panel’s digital passport is valid, its provenance is correct, and it matches the architect’s specification.” They then submit a “Work Verified” proof to the ledger.
    3. The Architect (Visionary Designs Inc.): An on-site representative from the architect’s firm performs a final quality assurance check. They, too, scan the panel. Their system performs the most critical check: it cross-references the panel’s digital passport against the original BPD specification they themselves lodged. If it matches, they submit the final “Specification Compliant” proof.

    How Cheating is Prevented Let’s imagine scum amongst the subcontractor, “GlassCurtain Facades,” attempts to cheat by installing the cheaper, flammable Chinese cladding. •

  • Scenario 1: The Fraudulent Panel has no Digital Passport.
    ◦ The installer’s device will immediately fail when they try to scan the panel. It has no valid QR code or NFC tag. They cannot create the “Proof of Installation.” The system halts. An alert is raised.
    • •
  • Scenario 2: Due to the Actions of Scum in the Supply Chain, the Fraudulent Panel has a Forged Digital Passport.
    ◦ The installer scans the panel and submits their “Proof of Installation.”
    ◦ The General Contractor’s device scans the panel. It queries the ledger for the passport and finds that the cryptographic signature from the original manufacturer (Reynobond in France) is missing or invalid. The check fails. The system halts. An alert is raised.
    ◦ Even if a scumbag contractor is complicit and submits a “Work Verified” proof, the system still requires the architect’s sign-off.
    The architect’s device will find that the passport, even if it looks legitimate, does not match the original specification they lodged. The check fails. The system halts. An alert is raised.
    • •
  • Scenario 3: Collusion.
    ◦ What if scum amongst the installer, the contractor, and the architect’s representative on-site all collude? The scumbags all agree to sign off on the fraudulent panel.
    ◦ This is where ChubbyChecker comes in. The Cross-Check “cc113_immutable-field-overwrite” would run against the transaction.
    It would compare the material_spec field from the original BPD with the material_installed field from the “Proof of Installation” transaction.
    They would not match.
    ◦ The check would fail, an evidence package would be broadcast to the entire federation, and the fraudulent activity of the scum would be exposed to all stakeholders in real-time, including the client and regulators.

    • The End Result The system makes cheating not just difficult, but computationally and logistically impractical. To succeed, the scum would need to:

    1. Forge a digital passport with a valid cryptographic signature from a reputable manufacturer.
    2. Bribe the on-site representatives of at least two other independent companies.
    3. Somehow circumvent the automated, independent checks being run by the ChubbyChecker system.

    The cost and complexity of the fraud would far outweigh the savings from using the cheaper material.

    The system doesn’t rely on human trust; it relies on cryptographic certainty and multi-party consensus.

    ________________________________________________________________________

    StormBringer

    ________________________________________________________________________

    One Bucordo node per member or member- class, with substrate database, the server running the GoldBFT (our fork of CometBFT with enhanced transports) GO package, each replicating from their own substrate servers to all other Bucordo nodes, which then each complete the replication, via their own GoldBFT applications (to the solo database server) in a 6 stage process. Communication is achieved via Notifications. (Replicating to a Bucordo node involves only notifications).

    Trust is policed programmatically as follows:

    1. The first stage is to cause ordering then execution of the forwarded Metadata from Customer-Enterprise-submitted transactions in Blocks. The transactions in the cases of Customers with Integration Modes B, C, D are records with 33 fields each as the standard Tx-Rails Metadata set. In the case of Mode A, a single Hash value is stored – one hash per Customer Enterprise transaction. This is the initial ordering/execution/replication process.
    2. The second stage is to commit or abort the same executed transactions, using a SSI (Serialisable Snapshot Isolation) heuristic to determine which transactions (if any) need to be aborted (rolled-back). Rolling back may be necessary due to threatened State-Inconsistency between transactions targeting the same table in a single block.
    3. Substrate Masters (“lagerhaus”) send their own State-Change Hashes to their own additional system tables that store brief details of Transactions and Transaction_Blocks. These Hashes must be identical across nodes. If a total of more than f nodes are affected (where f = (n – 1)/3, and n is the total number of nodes from the set {7, 10, 13}) by detected inconsistencies in the Hashes or in any of the other 132 items cross-checked, an alarm is broadcast to all Stakeholders. A Contingency Plan is implemented if an Alarm State exists.
    4. This State-Change Hash is added to the next Block, currently being formed, early in stage 2 above.
    5. The penultimate stage involves a Checkpointing process, which the IBM article indicates should be performed at less than the rate of once per Block-Committal (for efficient performance and optimum outcomes – mainly to minimise false positives), on the Substrate Masters, with an outcome of Rolling Back any remaining inconsistent Transactions. Inconsistent transactions at this stage raise a Warning, with Status to be confirmed at the next Ultimate check, which is broadcast to all stakeholders, and Contingency Plans are implemented. The detection of False positives is an essential part of the process at this stage.
    6. Before finalising the cycle, a Consistency Check ChubbyChecker is run against the Hedera Blockchain’s records of Transaction Traces. If Stage 5 above were completed without incident, but if the Ordering services had acted in collusion with the cooperation of > 33% of Sites, or if external hackers have been involved, in order to defraud the system, or other criminal scum attempted tampering, this situation would still not have been detected. To achieve the detection of such a situation, Merkle Roots of Transaction Traces are recorded as objects, or “Assets”, on the Hedera Blockchain, but full details of traces are recorded on WORM drives (databases which are “append only”), after the full Enterprise Transactions (which generate this metadata) have been executed on organisation databases. As each Block of Transactions is processed, replicated and then committed (or aborted), further Assets are added on the neighbouring nodes and, Block-by-Block, their Trace Merkle Roots are registered on Hedera, resulting in comprehensive lists of the Merkle Roots of 24 Blocks x 1024 individual Pre-registered Transaction Trace records per “Slab” of Blocks, ie 24 x 2 files/node x N(i) per block – the number of nodes in Installation [i] containing original Transaction Traces formed into Blocks, with identical Trace Orderings, as the other nodes in an installation replicate and order the original transactions. Anti-Fraud checking, will be driven by reviewing the records of the Assets (Merkle Roots of Transaction Traces) registered on the immutable Hedera Blockchain, with the intention of investigating the status of external and internal consistency of the Transaction Traces. The content of these objects will be influenced by the status of the recorded files, with fields for “executed”, and either “rolled-back” or “committed”, “amount”, Client details, “TxUUID”, “BlSeq” (k), etc. Notice that, across nodes, for a series of Assets with identical Original TxUUIDs, either |{executed tx}| = |{committed tx}| OR |{executed tx}| = |{rolled-back tx}|, or inconsistency exists. In all cases the length of the series with identical original TxUUIDs must equal the number of nodes in the installation being examined. Unique transaction traces, will be registered with unique TxUUIDs (driven by the Tx(UUID)s + the Node Ids generated by our own applications at Pre-Registration) at the different stages of their lives (exec, possibly rolled-back, probably committed) to form a series of records of the history of the uniquely identified transaction traces. There are also possible “False Positive” indications. The occurrence of False Negatives will be difficult to handle as this would mean there is no registered sign of problems in an installation despite the actual existence of problems. How could we tell? Please refer to the section below for a comprehensive list of Internal Consistency Checks. Overall the cross-checking relies on replays of the pre-registered traces to enforce external consistency.

    This completes the 6-stage production cycle, incorporating Fraud checks.

      1. When the databases are rolled out they will have a structure of schemata, tables and columns identical to each other, but also with an extra 6 new tables per installation (postgres_system.public.certs, postgres_system.public.blockstore, postgres_system.public.blocks_transactions, postgres_system.public.ledger_preregister, postgres_system.public.ledger_postregister_exec & postgres_system.public.ledger_postregister_commit), located in a postgres_system.public database/schema in the cluster. There will also be new “update_ledger_postregister_exec” & “update_ledger_postregister_commit” triggers (one of each per ordinary table throughout the databases) and trigger functions (one of each per ordinary schema in the databases), and the new “update_block_count” trigger and function (fired by the new postgres_system.public.ledger_postregister_commit table) targeting the new postgres_system.public.blockstore system table in the postgres database. The records in the postgres_system.public.ledger_postregister_exec & postgres_system.public.ledger_postregister_commit tables are updated by the firing of the “update_ledger_postregister_exec” & “update_ledger_postregister_commit” triggers. On the postgres_system.public.ledger_postregister_commit table, the “update_block_count” trigger only fires once upon committing every T unique transactions. Note that the Tx(UUID)s are created at Pre-Registration (by the receiving load balancers for the V.M.’s), and inserted into each query subsequently processed, leading to automatic inclusion of uuid’s upon replication. The transactions are identified by their multiply-copied originating TxUUID’s combined with the node ID upon which the current TxUUID is being replicated. When postgres_system.public.ledger_postregister_commit has counted ‘T’ unique transactions, (where unique TxUUID’s are being counted on postgres_system.public.ledger_postregister_commit as the ordinary schemata have been firing their own “update_ledger_postregister_exec” & “update_ledger_postregister_commit” triggers upon executing & committing the ordered transactions. Thus, there will be N(i) copies of each TxUUID in each of the 2 tables postgres_system.public.ledger_postregister_exec & postgres_system.public.ledger_postregister_commit – the number of nodes in the installation), “update_block_count” is fired and postgres_system.public.blockstore.BlSeq is incremented, along with updates of the other fields in postgres_system.public.blockstore, on behalf of all databases in the installation, and postgres_system.public.blockstore is locked until GoldBFT has committed the ordered Block. The information included in these records is drawn from postgres_system.public.ledger_postregister_commit’s own database records.
        Currently, T = 1024.
      2. As Blocks are committed, the “postgres_system.public.blockstore” table is keeping a cyclic count of committed Blocks (in the BlSeq field), ie being updated by firings of the update_block_count trigger on the new postgres_system.public.ledger_postregister_commit table, with an enum operating on “postgres_system.public.blockstore” which runs from 0 to K-1. When that postgres_system.public.blockstore.BlSeq enum reaches a count of K-1, the enum resets, and a Notification is broadcast to the Chubby-Post-Register and ChubbyChecker containers announcing the end of an Ultimate-Registration Phase (when the number of Blocks committed reaches ‘K’), thus signalling the commencement of the next ChubbyChecker round of checks. ChubbyChecker
        Currently, K = 24.
      3. With regard to the method of recording and examining the preregistered transaction states, the load balancer onboard each virtual machine, intercepts each transaction, passing the required metadata detail, encrypted, to the local ChubbyChecker-Joint server which writes this data to its database. Upon Transaction Execution and also upon Transaction Commitment/Roll-back, the lists of a. Executed. b. Committed and. c. Aborted Transaction Traces are written as a Block of details to the Aggregators by the Bundlers. At the Bundler there is a bifurcation generating a process which also passes these newly executed/committed/aborted Traces to ChubbyChecker-Joint. Separately, the Tophat servers receive similar data from the Aggregators via the Bundlers, examine the outputs for consistency against Hedera and PostGIS and if all is well, write the latest Merkle Roots to Hedera. Meanwhile, the relevant ChubbyChecker container reads the Bundler outputs after Commitments/Rollings-back of the current Block of transactions, and creates database entries against which it compares the executions/commitments/rollings-back with the preregistered Baseline (as intercepted by the load balancer). ChubbyChecker-Joint uses the list of transactions by UUID contained in the Block (and therefore the detail only summarised in the latest Merkle Roots on Hedera) to securely define the boundaries of one Block of Pre-Registered Transaction Traces, a posteriori.. This then allows the system to perform all 129 transaction trace cross checks on the current “Slab” of 24 Blocks of (maximum) 1024 transactions per Block. These checks are then completed after the Hedera Merkle Roots have been written, and the system therefore requires a File type on Hedera that would serve to mark any occasions of failed cross-checks which are not false positives. It is the responsibility of ITOTCCA’s ChubbyChecker to initiate (temporary) server shutdowns, if the number of faulty nodes exceeds f (see above), as any other pathway would be too risk-prone. Investigators are then convened.

    (thus implying that
    1. each Orderer Node has an Application Id &
    2. that the current Block has passed the Consistency checks (Process 6)).

    Following the lead of IBM, we are looking towards the Tendermint/CometBFT Byzantine Fault Tolerant Transaction Replication & Ordering Service, which is to be installed on each Peer Bucordo node, accepting input from the “gRPC-servlet” module on the Smartweb server. The Bucordo Ordering peers (BFT-SMaRt Client & Server) co-reside with the gRPC servlet, the Elastos Carrier/DID system and the Chubby-Post-Register and ChubbyChecker containers in several “pods” on a virtual machine altogether comprising one “node” when the database is included: there are 6 to 8 nodes per vm, and 4 vm’s on our development workstation.

    Replicação da máquina de estado bizantina para as massas

    Byzantine State Machine Replication for the Masses
    Embrittlement

    We initially assessed 11 major risks, augmented to 98 items associated with the bft-dht System. We have since extended this to a total of 132 unique checks (covering technical risks 1 through 132) in ‘ChubbyChecker’.
    xAt the completion of Registration of the Kth Block (ie at the end of each Ultimate Registration Phase), we begin the full set of 132 Checks:
    Check the following conditions are MET for at least n – f nodes (see above), else ALARM STATE:ChubbyChecker
    (Note that currently the number of Blocks
    in a single Ultimate Registration/Checking Phase, K = 24,
    and there are designed to be a maximum of 1024 Transactions per block)

    🧠 The Initial 11 Distributed Validation Failure Classes

    1. Weakness: Reliance on DNS
      2. Risk: Various Attack Vectors are enabled at every DNS lookup Mitigations: By using a Distributed Hash Table network, and the Kademlia Protocol, exclusive and secure Client/Bank communication is empowered. The system is an Absence of DNS system after the first connection occasion, or after a Node Id rotation.
    2. Weakness: Initial Trust Anchor Must Be Protected
      3. Risk: TLS bootstrapping relies on traditional PKI infrastructure. A compromised device or certificate store undermines security from the start. Mitigations: Certificate Pinning: Ensure the client only trusts a pre-defined certificate for the bank’s HTTPS bootstrap server. Hardware-Backed Key Stores: Leverage TPM/Secure Enclave to safeguard certificate validation processes. In-Person or Offline Bootstrap: Offer bootstrapping credentials (node list, hash pins) through physical branch or secure mail. Frequent Certificate Rotation: Reduce exposure window by rotating TLS certs (and corresponding pinned fingerprints) frequently and notifying users securely.
    3. Weakness: SQL Injection Attacks
      4. Risk: By crafting malicious queries, if direct SQL queries are allowed, antagonists can perpetrate Fraud and/or Damage Mitigations: Use only Prepared Statements in queries.
    4. Weakness: Sybil Attacks and Routing Vulnerability
      5. Risk: An attacker could flood the DHT with rogue nodes to intercept or misroute traffic. Mitigations: Node Whitelisting: Supernodes only accept known, cryptographically signed peer IDs. Node ID Signing Authority: Bank uses a CA to issue signed node identities for all its DHT participants. Periodic Identity Rotation: Nodes are periodically reissued new identities, preventing long-term accumulation of fake peers. Peer Behaviour Monitoring: Implement scoring and blacklisting mechanisms for abnormal routing or message behaviours.
    5. Weakness: Content Poisoning and Misrouting
      6. Risk: Old or tampered routing/content data could be served if not checked correctly. Mitigations: Content Signing: Every response from a DHT node must be cryptographically signed by the bank node issuing it. Versioned Metadata with Hash Validation: Store transaction metadata with SHA256 hashes and version stamps to verify integrity. Reject Unsigned or Invalid Data: Enforce signature validation on all received metadata and configuration payloads.
    6. Weakness: Node Churn and Reconnection Logic
      7. Risk: Clients may be unable to reconnect after a long absence if node IDs have changed or expired. Mitigations: Backup Bootstrap List: Provide multiple fallback bootstrap DHT node IDs. Out-of-Band Recovery: Users can receive an updated node list via secure email, physical mail, or authenticated mobile apps. Graceful Node Expiry: Allow a short grace window where old node IDs remain available in parallel with newer ones. Client Watchdog Mechanism: Alert users if the node list expires or is unreachable, prompting a safe refresh process.
    7. Weakness: Latency and Reliability Trade-offs
      8. Risk: DHTs introduce more latency than centralised systems and may suffer from network churn. Mitigations: Peer Clustering: Organise nodes into latency-aware geographic/organisational clusters to improve resolution speed. Caching with TTL: Cache verified metadata locally with proper TTL and freshness checks. Relay Nodes or Gateway Optimisation: Deploy performance-optimised supernodes or relay nodes for high- demand clients. Redundancy & Parallel Lookup: Send redundant queries across peers to ensure quick and correct responses.
    8. Weakness: Legal and Auditing Concerns
      9. Risk: Dynamic and decentralised routing can complicate regulatory compliance and forensics. Mitigations: Immutable Anchored Logs: Use Tendermint to write tamper-evident logs into PostGIS with unique identifiers per request. Metadata Snapshots: Include blockHeight, slabId, supernode-issued timestamp and Digital Client Signatures in every transaction trace. Auditable API Endpoints: Allow regulators to inspect transaction records via secure, read-only APIs with Merkle proof generation. Data Residency Configurations: Supernodes can be bound to specific jurisdictions to respect local storage regulations.
    9. Weakness: Client-Side Security Still a Weak Point
      10. Risk: Even the best backend doesn’t prevent malware, keyloggers, or compromised client devices. Mitigations: Transaction Signing with Secure Element: Offload all sensitive signing operations to a hardware-secured key store (e.g., Secure Enclave, TPM). Mobile Companion App: Use an independent device to sign transactions (QR-code pairing or push- approval). Behavioural Biometrics or MFA: Include biometric, geolocation, or behavioural analysis before permitting critical operations. Zero Trust Session Validation: Supernodes should perform heuristic checks (IP change, device ID mismatch, login anomaly).
    10. Weakness: Overcentralization of Bootstrapping Supernodes
      11. Risk: Though the system avoids DNS, if all bootstrapping, ie at very first step, routes through a single supernode or set of banks, it risks reintroducing a single point of failure or control. Mitigations: Federation Governance Charter: Define a decentralised multi-bank governance model for supernode interoperability. Public Key Transparency Logs: Maintain a tamper-proof record of all bootstrapping key changes or additions. Rotating Peer Authorities: Rotate trusted bootstrapping authorities or require multisig endorsement from a quorum of banks.
    11. Mitigate Additional Weakness_11: consists of the requirement that, Across all nodes, after each block of transaction commit procedures, all the State-Change Hashes recorded by all replicas (ie nodes) must be identical.
      12. grenfell2 grenfell3

      🧠 Additional 29 Distributed Validation Failure Classes

    12. Check 12: Node Identities: Verify that the nodes involved in the consensus process are the expected, authorized nodes.
    13. Check_13. Across all transactions, in Post-Registration-execute and in Post-Registration-commit/abort phases, block sequence numbers must be continuous.
    14. Check_14. In the Pre-Registration files, ostensibly conitnuous among the K Blocks, the Traces cannot be divided into Blocks a priori. However by writing a block marker file to the etcd/DHT system when each block (ie decided by machinic consensus) is denoted and recording the TxUUIDs contained in the block, and writing an encrypted file with that Smart Contract’s Id, locally (ie to the vm or “installation”), ChubbyChecker can assign traces, from the Preregistration Stage, to blocks, and check Block continuity a posteriori. Using this method, on each node, the total of Preregistered (submitted) Transactions in one Ultimate Checking Phase must equal (total of TxUUID’s executed) – in the Post-Registration-execute phase – AND must equal (total of TxUUID’s committed + total of TxUUID’s aborted) in the Post-Registration-commit/abort phase, across all Post-Registered Blocks in the same Checking Phase.
    15. Mitigate Additional Weakness_15: Across all nodes, and for both Post-Registration-execute and Post- Registration-commit/abort phases, for a set of transactions in BlockSeq (k), with TxUUID = z: ([Total of TxUUID’s with opCodeExec == “true”] == [Total of TxUUID’s with opCodeCommit == “true”] OR [Total of TxUUID’s with opCodeExec == “true”] == [Total of TxUUID’s with opCodeAbort == “true”]) AND len{TxUUID = z} = N(i), the Number of nodes in the installation [i] being examined , for all (k, z) in [i]. (Transactions in k determined a posteriori from phase (c), as in Mitigate Additional Weakness_14).
    16. Mitigate Additional Weakness_16: Block Hash Integrity: Validate that the block hash matches the expected value, confirming that the block has not been tampered with.
    17. Mitigate Additional Weakness_17: Transaction Hash Integrity: Verify the integrity of individual transactions within the block by checking their hashes.
    18. Mitigate Additional Weakness_18: State Consistency: Ensure that the state after each block execution is consistent across all nodes, reflecting correct and deterministic execution.
    19. Mitigate Additional Weakness_19: Commit Consistency: Confirm that the blocks committed by different nodes are identical, with no divergence in the Hedera Blockchain.
    20. Mitigate Additional Weakness_20: Byzantine Fault Detection: Analyze any discrepancies in node responses to detect possible Byzantine behavior.
    21. Mitigate Additional Weakness_21: Message Authenticity: Verify that all messages exchanged between nodes during the consensus process are authentic and correctly signed.
    22. Mitigate Additional Weakness_22: Network Partition Detection: Detect any network partitions or communication issues that might have influenced the consensus process.
    23. Mitigate Additional Weakness_23: Latency and Timing Consistency: Ensure that the timing of block commits and transaction execution is within expected bounds, identifying any anomalies that could indicate manipulation or delays.
    24. Mitigate Additional Weakness_24: Redundant Storage Consistency: Verify that redundant copies of transaction data and state information are consistent across nodes.
    25. Mitigate Additional Weakness_25: Recovery Point Integrity: Ensure that the recovery points or checkpoints created during the process are accurate and can be used to restore the system state if needed.
    26. Mitigate Additional Weakness_26: Across all checked transactions (Pre-Registration-submit, Post- Registration-execute and Post-Registration-commit/abort phases), each signature is traceable to a current client from the current installation’s range of member-class servers’ (ie nodes’) clients.
    27. Mitigate Additional Weakness_27: Across all checked transactions, the ‘entnum’ field in the transaction’s contents, when compared to the client’s registered ‘entnum’, must match.
    28. Mitigate Additional Weakness_28: Across all checked transactions, the ‘schema’ field in the transaction’s contents, when compared to the client’s available ‘schemata’, must match one item. These are fully qualified names ie database.schema.table.
    29. Check 29. The “amount” field in all financial transactions, and where amounts exist, in any other transaction, are constant across preregistration (submission), execution and committal/roll-back.
    30. Check_30. On each node, TxUUIDs are generated at Pre-Registration, so in each of the k BlockSeq numbers, for (a). Pre-Registration-execute phase, (b). Post-Registration-execute phase and (c). Post-Registration-commit/abort phase, each TxUUID occurs once exactly, and the TxUUIDs in (a) are “1 to 1” AND “onto” the sets for (b) and (c). See Check_3 above for method of distinguishing blocks.
    31. Check_31.On each node, across phases (b) and (c), [Total of TxUUID’s with opCodeExec == “true” (which should be all TxUUID’s)] == [Total TxUUID’s with opCodeCommit == “true”] + [Total TxUUID’s with opCodeAbort == “true”].
    32. Check_32. On each node, the total number of structs in the array of structs, spanning each BlockSeq (k), is equal to the agreed number (eg 1024) of transactions per Block, and this holds for all of (a). Pre-Registration-execute, (b). Post-Registration-execute, and (c). Post-Registration-commit/abort phases. Here the number of transactions in each BlockSeq for phases (a) and (b) is determined a posteriori by reference to phase (c), as in Check_3.
    33. Check_33. Across all nodes, at the Post-Registration-commit/abort phase, for a full set of transactions in BlockSeq (k): the ordering of transactions must be constant for all member class servers (nodes), and for all k under examination.
    34. Check_34.The value of the previous Transaction Trace Merkle Root is retrieved from the Smart Contract and compared to the value recorded on PostGIS. The values must be identical.
    35. Check_35.We then rebuild the Merkle Tree for the previous slab using the transaction trace database’s history and compare the rebuilt root with the one retrieved from the blockchain. If the rebuilt root differs from the blockchain root, it could indicate tampering in the underlying transaction data. This approach verifies not only the stored root but also the integrity of the transactions that contributed to it.
    36. Check_36.We validate that every transaction contributing to the current Merkle Tree has a corresponding entry in the metadata store. We ensure that these entries match the expected data (e.g. transaction IDs, hash values).
    37. Check_37.Instead of relying solely on the Merkle Root, we include “hashes” of individual transactions in the leaves of the Merkle Tree. This ensures that even if the Merkle Root is consistent, individual tampered transactions can be identified by checking their hashes.
    38. Check_38.We use periodic snapshots of the database, stored securely in an independent location, to cross-validate the database state. Snapshots are used to detect discrepancies that might otherwise go unnoticed due to tampering with live data. Thus we take database snapshots at the end of each slab. These snapshots are securely stored using immutable storage. We use the snapshots to rebuild and validate historical Merkle Trees
    39. Check_39.Proof-of-Consistency algorithms (like those used in blockchain systems) are used to ensure that:

      The current Merkle Root is consistent with the previously recorded root.

      No invalid transactions have been added between slabs.

    40. Check_40: consists of the requirement that, Across all nodes, after each block of transaction commit procedures, all the State-Change Hashes recorded by all replicas (ie nodes) must be identical.
      41. bank

      🧠 There are 32 Business Process Definition and Governance
      Validation Failure Classes

      🧠 74 further Common (and less common)
      Distributed Validation Failure Classes

    41. Mitigate Weakness_41: Commit Consistency:

      Confirm that the blocks committed by different nodes are identical, with no divergence in the blockchain.

    42. Mitigate Weakness_42: Latency and Timing Consistency:

      Ensure that the timing of block commits and transaction execution is within expected bounds, identifying any anomalies that could indicate manipulation or delays.

    43. Mitigate Weakness_43: Recovery Point Integrity:

      Ensure that the recovery points or checkpoints created during the process are accurate and can be used to restore the system state if needed.

    44. Mitigate Weakness_44: Client Signature Traceability

      (Pre-Registration-submit, Post- Registration-execute and Post-Registration-commit/abort phases), each signature is traceable to a current client from the current installation’s range of member-class servers’ (ie nodes’) clients.

    45. Mitigate Weakness_45: Client Entnum consistency,

      the ‘entnum’ field in the transaction’s contents, when compared to the client’s registered ‘entnum’, must match.

    46. Mitigate Weakness_46: Client Schema consistency,

      the ‘schema’ field in the transaction’s contents, when compared to the client’s available ‘schemata’, must match one item. These are fully qualified names ie database.schema.table.

    47. Mitigate Weakness_47: Phantom Write

      A transaction appears to commit, but state is not reflected on all nodes. Detected by: Interceptor’s pre-block record vs post-slab snapshot Missing keys/rows in PostGIS after commit

    48. Mitigate Weakness_48: Lost Rollback

      A rollback request was issued but at least one node still applied the transaction. Detected by: Interceptor snapshot shows txn = pending, slab shows txn = applied Mismatch in rollback marker across nodes

    49. Mitigate Weakness_49: Out-of-Order Commit

      Transaction T2 committed before T1, violating causality or application-level ordering. Detected by: Timestamp or block height comparison Violation of vector clock or logical clock constraints

    50. Mitigate Weakness_50: Replica Divergence

      Two or more nodes apply the same transaction with different outcomes (e.g., amounts, timestamps). Detected by: Node-wise comparison of trace outcomes Hash/Merkle mismatch across replicas

    51. Mitigate Weakness_51: Replay Detection Failure

      A previously committed transaction is accepted again as if new (e.g., due to hash collision, retry abuse). Detected by: Duplicate transaction hash in slab Non-monotonic nonce reuse

    52. Mitigate Weakness_52: Inter-slab Drift

      • A transaction committed in slab N is reflected differently in slab N+1 (e.g., metadata vs actual state). Detected by: Metadata slab summary mismatch Commit cross-reference checks (super- checker logic)

    53. Mitigate Weakness_53: Transaction Gap

      • Missing transaction in the expected slab range (e.g., slabs show txns 100–124, but 117 is missing). Detected by: Monotonic counter comparison Range scans with missing indices

    54. Mitigate Weakness_54: Clock Skew Violation

      • A node reports a commit time outside slab boundaries or before it saw prior events. Detected by: Comparing slab timestamps and txn timestamps across nodes Logical vs wall clock drift detection

    55. Mitigate Weakness_55: Stale Read/Write

      • A node uses a version of data that’s behind consensus state (write skew, old snapshot). Detected by: Comparing pre-block data with committed data Cross-checking snapshot IDs or transaction IDs used during execution

    56. Mitigate Weakness_56: Write Amplification Error

      • A single transaction causes multiple unintended writes (e.g., duplicate rows, unintended triggers). Detected by: PostGIS trace diff exceeds expected write footprint Interceptor baseline vs actual write set

    57. Mitigate Weakness_57: Partial Metadata Commit

      • A transaction’s metadata was committed, but its payload wasn’t (or vice versa). Detected by: Comparing metadata slab to enterprise slab Supernode-specific slab parity checks

    58. Mitigate Weakness_58: Invalid Trace Termination

      • A transaction trace does not reach a terminal state (commit, abort, etc.). Detected by: Orphaned trace entries in slab Missing status = final marker in logs

    59. Mitigate Weakness_59: Ghost Transaction

      • A transaction exists in traces but not in slab metadata or committed state. Detected by: Slab metadata doesn’t account for an otherwise traceable transaction Missing block or tx hash in ledger indexes

    60. Mitigate Weakness_60: Misrouted Transaction

      • A transaction appears on a node not responsible for its class or shard. Detected by: Node role mismatch vs transaction class Supernode incorrectly applying leaf-level transactions

    61. Mitigate Weakness_61: State Bleed

      • Data from a previous transaction or node incorrectly appears in a new one. Detected by: Unexpected field values or foreign keys Orphaned joins across slabs

    62. Mitigate Weakness_62: Metadata Omission

      • Metadata slab lacks traceable linkage to underlying enterprise transactions. Detected by: No matching slab ID, txn ID, or hash from metadata to main trace Hash chain break in Merkle anchoring • Data from a previous transaction or node incorrectly appears in a new one. Detected by: Unexpected field values or foreign keys Orphaned joins across slabs

    63. Mitigate Weakness_63: Incomplete Propagation

      A valid transaction is present on some nodes but entirely missing on others. Detected by: gRPC/QUIC checker receives “unknown transaction” errors from subset of nodes Missing consensus message on affected replicas

    64. Mitigate Weakness_64: Signature Mismatch

      Digital signature or identity on a transaction is invalid or unverifiable. Detected by: Signature fails verification against stored pubkey Signature valid but from unexpected signer

    65. Mitigate Weakness_65: Block Hash Mismatch

      • A slab’s hash root differs between nodes for the same slab ID. Detected by: Merkle root or hash mismatch in slab headers Inconsistent metadata anchoring

    66. Mitigate Weakness_66: Cyclic Transaction Dependency

      • A transaction depends on another transaction that, transitively, depends back on it. Detected by: Dependency graph cycle in trace metadata Impossible causality

    67. Mitigate Weakness_67: Orphaned Commit

      • A commit was logged, but no corresponding execution trace or context exists. Detected by: status=committed, but missing execute or prepare phase in trace Unexpected commit timestamp with null origin

    68. Mitigate Weakness_68: Multi-Node Conflict

      • Two nodes report different accepted transactions for the same logical slot. Detected by: Slot- based ledger conflict Concurrent append without conflict resolution

    69. Mitigate Weakness_69: Nonce Reuse

      • A client or process reused a nonce or unique identifier, resulting in conflicting operations. Detected by: Multiple transactions with identical nonces and different contents Consensus dispute on transaction ID uniqueness

    70. Mitigate Weakness_70: State Lag

      • A node accepts and executes a transaction using stale global state. snapshot/version ID Prior commit not visible during execution

    71. Mitigate Weakness_71: Inconsistent Rollback Outcome

      • Rollback succeeded on some nodes but left partial effects elsewhere. Detected by: Stale Detected by: Residual entries in PostGIS from rolled-back transactions Uneven rollback markers across replicas

    72. Mitigate Weakness_72: Anomaly Window Leak

      • A transaction that should not be visible yet is observable by another in the same slab. by: Temporal ordering violations between transactions Snapshot isolation breach

    73. Mitigate Weakness_73: • Cross-Shard Drift

      A cross-shard transaction commits parts in different slabs (or commits in one, not another). Mismatched slab IDs for shard-aligned operations Atomic cross-shard violation

    74. Mitigate Weakness_74: Audit Trail Omission

      • A transaction completed, but audit logging or trace proof is missing or incomplete. Detected Detected by: Detected by: Lack of external anchoring (e.g., Hedera hash) Empty or truncated audit trail entries in metadata

    75. Mitigate Weakness_75: Transport-Level Mismatch

      • A transaction hash or slab root differs in transport versus what the node actually committed. Detected by: Comparing gRPC/QUIC pushed headers vs actual PostGIS state Ensuring message headers don’t mask node-specific drift
      76. logs

    76. Mitigate Weakness_76: Double Acknowledgment

      • A transaction is acknowledged as committed twice with different metadata (timestamp, trace ID). Detected by: Conflicting ACK or commit records in metadata slabs Mismatch in quorum acknowledgement count

    77. 📉 Mitigate Weakness_77. Failed Rollback with Success Signal

      Node reports rollback success but internal state still mutated. 🔍 Detected by: Compare rollback trace with actual PostGIS record Trace ends in “rolled back” but values differ from original

    78. Mitigate Weakness_78: Log Rewind or Overwrite

      • Node replays or overwrites log entries without emitting a slab boundary. Detected by: Sequence ID reversal in logs or non-monotonic slab offsets Unexpected gap or repeat in block IDs

    79. Mitigate Weakness_79: Query Path Drift

      • Identical queries return different results across nodes for the same slab. Detected by: Compare SELECT results per node post-check Look for index path inconsistencies or filters applied differently

    80. Mitigate Weakness_80 Slab Compression Divergence

      • Slab compression or delta encoding introduces non-determinism across nodes. Detected by: Hash mismatch after decompression Bitwise diff on slab archives

    81. Mitigate Weakness_81: Anchoring Drift

      • The slab anchoring (e.g., to Hedera) differs in hash or timestamp from the actual slab content. Detected by: Hash anchor vs recomputed Merkle root of slab Anchored timestamp older than actual slab write

    82. Mitigate Weakness_82 Transaction Range Overlap

      • Slabs overlap in transaction ranges due to misaligned boundary commits. Detected by: Block N shows transactions 100–124; Block N+1 starts at 120 Double application or accidental replay of trailing txns

    83. 🐢 Mitigate Weakness_83. Slow Commit Visibility

      Slab committed but one or more nodes remain unaware past acceptable delay. 🔍 Detected by: Slab version visible on node A but unknown on node B after timeout Exceeds MaxSlabDrift threshold

    84. Mitigate Weakness_84: Persistence Loss

      • A slab was committed, but lost from persistent storage on restart or crash. Detected by: Slab metadata exists in memory, missing from disk query lsn mismatch on restart

    85. Mitigate Weakness_85: Schema Drift Between Nodes

      • One or more nodes apply schema changes (DDL) inconsistently across slabs. Detected by: Comparing pg_attribute, pg_class, or hash of information_schema Slab trace references column not present in schema

    86. Mitigate Weakness_86: Redundant Slab Execution

      • A slab’s transactions are re-applied after already having been committed. Detected by: Slab ID reuse in logs Double-application markers (e.g., timestamps increase, data unchanged)

    87. Mitigate Weakness_87: Transaction ID Reordering

      • Global transaction ID sequencing is violated across nodes. Detected by: Non-monotonic txID comparison between nodes Reordered slab input despite consistent timestamps

    88. Mitigate Weakness_88: Slab Freezing Without Finalization

      • A slab stops receiving transactions but never emits a finalized or anchored state. Detected by: Slab length == 24, but no commit/anchor entry Trace exists, but slab footer missing or corrupted

    89. Mitigate Weakness_89: Partial Crash Recovery

      • Node restarts mid-slab but applies only a subset of the original transactions. Detected by: Fewer rows in slab N on one node vs others Gaps in LSN or checkpoint mismatch after restart

    90. Mitigate Weakness_90: Interceptor State Desync

      • The interceptor captured pre-block trace but missed or misaligned commit result. Detected by: Interceptor trace exists without matching committed trace Pre-block txnID has no post-commit pair

    91. Mitigate Weakness_91: Inverted Causality via Metadata

      • Metadata slab reflects future state inconsistent with current slab content. Detected by: Metadata slab shows post-commit results before actual data slab is committed Cross-slab hash references violate forward-only progression

    92. Mitigate Weakness_92: Multi-Stage Transaction Collapse

      • Multi-stage transactions (e.g., prepare → commit) collapse to a single action without intermediate state. Detected by: Lack of prepare or validation entries Atomic txn seen only at commit with no earlier context

    93. Mitigate Weakness_93: Metrics/Telemetry Mismatch

      • System telemetry reports successful execution inconsistent with actual slab content. Detected by: Prometheus/gRPC reports txn OK, but not in slab Metrics count vs slab tx count mismatch

    94. Mitigate Weakness_94: Immutable Slab Mutation

      • A slab marked immutable (finalized, anchored) is altered in subsequent traces. Detected by: Slab hash differs from previously anchored hash gRPC or PostGIS responses differ over time for slab ID

    95. Mitigate Weakness_95: Slab Boundary Drift

      • The slab boundary (24 txns) is not consistently respected, causing overflow or underflow. Detected by: Node slab has 26 or 22 transactions Metadata contradicts boundary definitions

    96. Mitigate Weakness_96: Inconsistent Autogeneration

      • Auto-generated values (e.g., UUIDs, serials) differ for the same transaction across nodes. Detected by: UUID mismatch for rows with identical trace ID Sequence counters desynchronized between nodes

    97. Mitigate Weakness_97: Missing Event Propagation

      • A slab-triggered event (e.g., a downstream signal or webhook) is emitted by some nodes, not others. Detected by: Observed webhook receipts or logs vs actual slab commits gRPC downstream calls missing for valid slab

    98. Mitigate Weakness_98: Transaction Alias Collision

      • Transactions with different intents share the same alias or application-level ID. Detected by: Alias → txnID mapping is 1→N when it should be 1→1 Same alias used in conflicting contexts across nodes

    99. Mitigate Weakness_99: Ledger Height Skew

      • Nodes disagree on what block height a transaction or slab resides in. Detected by: Conflicting height metadata across slabs Merkle roots differ due to offset placement

    100. Mitigate Weakness_100: Invalid Cross-Entity Reference

      • A transaction links entities (accounts, records) that never co-existed in valid state. Detected by: Join trace shows impossible lifecycle overlap Temporal windows for referenced entities don’t align

    101. Mitigate Weakness_101: Anchor Reorg Violation

      • A previously anchored slab is altered due to ledger “reorg” or anchoring change without traceability. Detected by: Hedera/anchoring mismatch over time Anchored slab hash mutation or rollback

    102. Mitigate Weakness_102: Checksum Drift Between Redundant Stores

      A slab stored in both PostGIS and another mirror (e.g., S3, IPFS) yields different checksums. Detected by: Slab hash or canonical serialization differs between stores Compression, encoding, or serialization logic diverges

    103. Mitigate Weakness_103: Trace Count Disagreement

      The number of trace records per transaction differs across nodes. Detected by: One node shows 6 trace entries, another 5, for the same txn Missing intermediate state logs (e.g., prepare, validate)

    104. Mitigate Weakness_104: Stale Recovery View After Recovery

      , a node reports slab data that is provably older than committed state. Detected by: Restart causes node to expose pre-commit view Node rejoins with slab hashes not matching current state

    105. Mitigate Weakness_105: Behavioral Drift Between Node Versions

      Nodes running different logic versions diverge on transaction execution or metadata inclusion. by: Version tags differ in trace headers Execution outcome differs on same inputs

    106. 🏁 106. Replay Window Violation

      A replay protection mechanism accepts a transaction outside its valid nonce/timestamp window. 🔍 Detected by: Transaction accepted far outside expected slab time Same txn ID replayed without triggering conflict

    107. Mitigate Weakness_107: Context-Detached Execution

      A transaction executes without its required context (e.g., missing authorization, state binding). Detected by: Context reference ID present but data not loaded Null context fields on node A, populated on node B

    108. Mitigate Weakness_108: Unconsumed Compensation Logic

      A rollback triggers compensation logic, but it’s ignored or never emitted. Detected by: Rollback triggers in trace, but no matching compensation row Post-state inconsistent with rollback plan

    109. Mitigate Weakness_109: Immutable Event Re-Emission

      An event marked immutable (e.g., external webhook or anchor) is emitted again under new data. Detected by: Event ID reused with new payload Immutable status tag violated

    110. Mitigate Weakness_110: Slab Interleaving with External System

      Slab timing intersects or overlaps with external system epochs incorrectly. Detected by: Slab timestamp range crosses time boundary expected to isolate Out-of-order events between slab and blockchain anchor

    111. Mitigate Weakness_111: Inverted Entitlement

      A node without permission executes or includes a transaction it should not. Detected by: Role-check fails on trace but transaction accepted Write by unauthorized class (e.g., peer writes supernode data)

    112. Mitigate Weakness_112: Partial Merkle Proof Failure

      A node includes only part of a Merkle proof chain, or reuses proofs across slabs. incomplete, hash path mismatch Proof reused when recomputation expected

    113. Mitigate Weakness_113: Immutable Field Overwrite

      A field marked immutable by schema or trace constraints is altered post-commit. multiple values across slabs for same txn Immutable tag present in slab schema

    114. 🛑 Mitigate Weakness_114. Commit Without Quorum Inclusion

      A transaction is marked committed but lacks quorum evidence or BFT proof. 🔍 Detected by: No quorum round data or consensus record BFT logs missing for slab, commit hash not reconstructable

    As soon as these checks reveal positive Faults on more than f nodes, an Alarm is multicast, whereupon the databases in the offended installation are shutdown gracefully, IoT operations for the relevant installation(s) are overridden and set to “Operation without Database”, and then the affected databases are restarted in ‘Investigation’ mode, with ordinary users and operators excluded.

    All the preceding checks have been implemented as GO! functions within the ChubbyChecker package, without full cooperation of the database operations, until after completion of source code alterations.

    The “Das_Fuhrwerk” server(s) in the offended installation(s) is the basis from which we conduct investigations (it has an ‘investigations’ table). It falls to Executive Directors and the CEO, CFO and CIO to conduct the investigations. Database, ChubbyChecker and ChubbyRegister log files are examined. The alarm will have been issued with a Code attached. The meaning of the code accompanies the announcement, and will correspond to one or more of the checks outlined above. It is not possible, obviously, to predict the course and duration of the Root Cause Analysis.

    The idea is to utilise the initial buffered command line input prompts, in main.go, for trusted staff to issue Status Update broadcasts as the investigation proceeds. All data entered into the Investigations Table is reviewed by ITOTCCA directors and is made available to Officers of Companies involved in the offended Installation(s), to check transparently the logic of conclusions drawn and evidence provided. Note that the first line of examination attempts to define whether the fault constitutes a “False Positive”.

    Normally, only 1 installation would be in an “offended” state at any time, however we do not rely on this. Multicasts are restricted to officers of companies within offending/offended installations.

    ________________________________________________________________________

    Tx-Rails — Built by IT/OT Chain & Cloud Australia (ITOTCCA). Sydney · London.