BFT/DHT ^Bucorde _{security worthy of} ^{Air Commodores}

________________________________________________________________________

‘BFT/DHT Bucorde’™

Twinned Networks

An Absence of DNS (ADNS) Enterprise System

Bucordo/BFT-SMaRt
Database and Server Nodes

In the DataCentres

[Supernodes, Secondary Nodes]

Kademlia/jvm-libp2p/DHT
Distributed Hash Table Nodes

Global

[Supernodes, Secondary Nodes,
Tertiary (Client) Nodes]

VS

With ChubbyChecker
And SuperChecker
Ultimate Transaction
Cross-Checking

“ChubbyChecker“.

QUIC/gRPC
HTTP3
Transport
Layer
Peer Discovery
by NodeId

TLS 1.3
Transport
Encryption
Total Absence
of DNS
~ ADNS ~

________________________________________________________________________

Merging adapted jvm-libp2p SuperNodes
with Bucordo PostGIS Database Nodes
Anchored to Blockchain

© by IT/OT Chain & Cloud Australia Pty Ltd 2022-2025

You might call it ‘100% FAULT INTOLERANCE for Databases’
(with In-Transit Security)

________________________________________________________________________

We get a first-pass level of security with BFT-SMaRt, which we strengthen using a “twinned network” in the form of an enhanced Carrier2 DHT Network overlaid on the BFT Network of Enterprise Databases (Secondary Nodes), with each Enterprise Network headed by one Carrier supernode per Business Network. All Carrier Supernodes participate in a separate Cluster-Network of Supernodes, where the metadata on this sub-net is also ordered and replicated by BFT-SMaRt. The network of supernodes controls the recording

of transaction trace metadata for the networks’ ChubbyCheckers, and is itself checked by SuperChecker, the top-level Checker.Supernodes play a role of interconnection between Wide Area separate clusters any time it is desirable to provide communication between these clusters and they are parts of a unified Enterprise Network. This enables intersite collaboration. All secondary node networks and the supernode network, record, and sequentially compare, their own Merkle Roots on Elastos.

ITOTCCA can provide Blockchain-networked Secure Database Access systems on the Elastos Smart Web, which largely automate intra-network trading and leave Banking, Email and and other traditional webservices to the traditional applications. However, at negotiable rates, we can advise on and implement government- and/or vendor-agreement-backed interfaces or workaround methods to save data in files, from Emails, Bank Statements and other traditional Web Services for validated reading into our ultra-secure systems, all initiated by you.

As mentioned elsewhere, all but the minimum necessary data to provide our security services and to manage the system remains secret from us and others. You value your Information. We respect that – and we provide our services with Privacy and Security as our highest priorities. The Metadata you “channel” to us, via your devices, automatically using our Public Key for encryption, and sent to our Metadata Stores, thence Anchored as a Trace of Traces to Elastos, amounts to a list of 20 data fields as follows:

Enterprise Number (entnum – Unique 64-bit Integer)

MemberOrClass Id (memberOrClassId – 28-bit byte array for the network’s members/member classes)

Key (recordId – Unique 100-bit byte array)

TransactionId (txId – UUID String)

BusinessNetwork (network – 28-bit byte array)

InstallationOrVM (installation – Big Integer)

Database (database – 98-bit byte array)

Schema (schema – 100-bit byte array)

Table (table – 100-bit byte array.)

Entire Tx Trace Hash (hash – 64-Byte Array (Merkle Leaf))

ClientId (clientId – Unique 44 character Base 58 array plus first 16 characters = ’09-0000000-0000000′. 60 characters sha3-512 hashed.)

TxSignStr (txSignStr – String)

BlockId (blockId – UUID String)

SlabId (slabId – Big Integer)

BlockHeight (blockHeight Small Integer)

Amounts (int64Amounts – List of Big Integers [JSONB])

Exponents (int32Exponents – List of Integers [JSONB])

OpCodeExec (opCodeExec – Boolean)

OpCodeCommit (opCodeCommit – Boolean)

MerkleRootHex (merkleRootHex) – Hexadecimal String)

Each Enterprise Transaction produces these 20 fields of Transaction Trace Metadata, multiplied by 1024 transactions per block per network then by 24 blocks per slab per network.
In addition to these fields, any client data which is indicated by any of the following conditions will likewise be encoded with ITOTCCA’s Public Key, processed then encrypted with the Customer’s Public Key before storage.

Data Types Requiring Decryption

Spatial Data: Geometries and geospatial properties.
Metadata: Transactional and spatial metadata.
Non-Spatial Data: Attributes linked to spatial entities.
Logs and Results: Audit logs, processed query results.
Data involved in Systems Management.

Operations Requiring Decryption

Spatial queries and transformations (e.g., ST_Within, ST_Buffer).
Joins, aggregations, and proximity calculations.
Validation, filtering, and indexing operations.
Processing and formatting results for client delivery.
Operations involved in systems management.

ALL DATA IS ENCRYPTED BOTH IN TRANSIT AND AT REST

GENERALLY USING YOUR OWN PRIVATE KEYS, BUT USING OUR PUBLIC KEY,

OR PARTNERS’ PUBLIC KEYS, TO SEND US CHANNELED, PRIVATE COMMUNICATIONS

Your set of cryptographic keys remains yours to manage with utter secrecy for different roles and users. We are not privy to customer crypto keys, with the exception of the common, business sub-network-wide, shared list of member public keys, intended to enable channeled secure and private communication.

Our system employs arrays of client-facing tables (with Table.System.oid mapping to UseCase.Id) corresponding to transaction use cases. Their data fields are designed to collect all client form-data followed by procedures involving trigger functions that process the data and store it in the Enterprise/Accounting-Structured parts of the databases. As such, one transaction will give rise to multiple records and record Ids, aside from the Record Id (RecordId_prelim) created in the original data-receiving table. All Enterprise databases/schemas have 2 sections 1. Class ‘A’ tables: Enterprise/Accounting section, and 2. Class ‘U’ tables: Use Case section. The Use Case Section is triggered to process and distribute data into the Accounting Section. In addition there are the DHT-connected Metadata Database and Storage Systems (PostGis on Supernodes with etcd medium-term persistence on the Secondary Nodes).

The following table is a summary of our 512-bit (Class ‘A’), 512-bit (Class ‘UR’), 362-bit (Class ‘U’) and special Base 58 encoded (an exception, with 44 Base58-encoded digits representing submitted Elastos Client DIDs) key systems (8 leftmost bits to signify Key Type, 28 bits for enterprise network enumeration, followed by 28 bits reserved for Member Class enumeration (when used), which are followed by unique 448-bit, or 298-bit Suffix sequences). Client Ids, as noted, represent an exception to these rules.
sha3-512 and reduced Key divisions:

1. KeyType=00000000₂ Peer/Node.Id
(Supernodes). Node Identifier Suffix: 448 bits:
Unique identifier. Full key prefix in Hex:
0x00-NNNNNNN-MMMMMMM

2. KeyType=00000001₂ Peer/Node.Id
(Secondary nodes). Node Identifier Suffix: 448 bits:
Unique identifier. Full key prefix in Hex:
0x01-NNNNNNN-MMMMMMM

3. KeyType=00000010₂ Peer/Node.Id
(Tertiary nodes). Node Identifier Suffix: 448 bits:
Unique identifier. Full key prefix in Hex:
0x02-NNNNNNN-MMMMMMM

4. KeyType=00000011₂ BFT-Database.Id
[including main Enterprise
Accounting Datatable Structure.] Identifier Suffix: 98 bits:
Unique general database identifier. Full key prefix in Hex:
0x03-NNNNNNN-MMMMMMM

5. KeyType=00000100₂ BFT-Schema.Id
[including main Enterprise
Accounting Datatable Structure.] Identifier Suffix: 100 bits:
Unique general schema identifier. Full key prefix in Hex:
0x04-NNNNNNN-MMMMMMM

6. KeyType=00000101₂ BFT-Class ‘A’
BFT-Table.Id
[including main Enterprise
Accounting Datatable Structure.] Identifier Suffix: 100 bits:
Unique identifier. Full key prefix in Hex:
0x05-NNNNNNN-MMMMMMM

7. KeyType=00000110₂ BFT-Class ‘A’
BFT-Record.Id
[including main Enterprise
Accounting Datatable Structure.] Identifier Suffix: 150 bits:
Unique identifier. Full key prefix in Hex:
0x06-NNNNNNN-MMMMMMM

8. KeyType=0000111₂ BFT-Class ‘U’
Database.Schema.Table
Corresponding to:
DHT-Database.
(public)Schema.Table.Record
for DHT_TX_Builder.
(Full Suffix: 298 bits) BFT-Table.Identifier (Trailing 100 bits)
of BFT Class ‘U’ table:
Identifies the enterprise database table: corresponding,
in addition, to Metadata Record Identifier
for Transaction-Builder
Record.Id (100 bits)
in DHT System.
The Full 298-bit key-suffix of the
user’s target client-facing
data-receiving Table in
BFT segment is used to
convert that BFT Table Id,
by prepending “00001000₂“,
to DHT-table-key suffixes
with leading Network
and MemberOrClass Ids
in the DHT system’s
Transaction Builder databases,
thus converting a Class ‘U’
BFT-system Table Id to a
Class ‘U’ DHT-system Record
Id, of the form DHT-Database.(publicSchema).Table.
Record Id, giving the key to
the required “Build Metadata” in the DHT System
for the current Request.] Full key prefix in Hex:
0x07-NNNNNNN-MMMMMMM

9. KeyType=00001000₂ BFT Class ‘UR’
Tx Trace Metadata Source

(Suffix: 448 bits) Table.Identifier (100 bits):
Identifies the Tx-Trace metadata table. Full key prefix in Hex: 0x08-NNNNNNN-MMMMMMM

10. KeyType=00001001₂ BFT-Class ‘UR’: BFT Enterprise Database.Id
(Suffix: 448 bits)
Associated, as Source Transactions,
with Transaction Trace Metadata
(DHT Class ‘UR’ Database) Record.Identifier (100 bits): Identifies the record. Full key prefix in Hex:
0x09-NNNNNNN-MMMMMMM

11. KeyType=00001010₂ DHT-Class ‘UR’: DHT Enterprise Database.Id
(Suffix: 448 bits)
Associated, with Source Transactions,
ie Transaction Trace Metadata
(DHT Class ‘UR’ Database) Table.Identifier (100 bits): Identifies the table. Full key prefix in Hex:
0x0A-NNNNNNN-MMMMMMM

12. KeyType=00001011₂ DHT Class ‘UR’ Database.Schema.
Table.Record.Id:
(Suffix: 448 bits) Record.Identifier (150 bits): Identifies the record.
Associated with BFT-Class ‘UR’:
BFT Enterprise Database.
This is the Transaction Trace
Storage System. Full key prefix in Hex:
0x0B-NNNNNNN-MMMMMMM

13. KeyType=00001100₂ DHT Class ‘U’ Database.Schema.
Table.Record.Id:
(Suffix: 448 bits) Table.Identifier (100 bits): Identifies the record.
Associated with BFT-Class ‘U’:
BFT Enterprise Database.
This is the Transaction Builder
Storage System. Full key prefix in Hex:
0x0C-NNNNNNN-MMMMMMM

14. KeyType=00001101₂ DHT Class ‘U’ Database.Schema.
Table.Record.Id:
((Suffix: 448 bits) Globally unique identifier for a record. Full key prefix in Hex:
0x0D-NNNNNNN-MMMMMMM

15. KeyType=00001110₂ Role.Id
(Suffix: 448 bits) Globally unique identifier for a Role. Full key prefix in Hex:
0x0E-NNNNNNN-MMMMMMM

16. KeyType=00001111₂ Client.Id
Personal Identifier
(Suffix: 448 bits) Globally unique identifier
for a client (personal).
Sha3-512 hash of client’s
Elastos DID with
Prefix (Hex) replacing
first 16 hashed hex digits. Full key prefix in Hex:
0x0F-0000000-0000000

17. KeyType=00010000₂ Value Identifier
(Suffix: 448 bits) Unique identifier for DHT values. Full key prefix in Hex:
0x10-NNNNNNN-MMMMMMM
18. KeyType=00010001₂ FULL_CLASS_A_KEY
(Suffix: 448 bits) 512-bit Key (Database + Schema + Table + Record) =
(fullTableName.Record).Id =
Db.Sch.Tab.Record Full key prefix in Hex:
0x11-NNNNNNN-MMMMMMM

19. KeyType=00010010₂ FULL_BFT-CLASS_U_KEY
(Suffix: 298 bits) 362-bit Key (Database + Schema + Table) =
(fullTableName).Id =
(Db.Sch.Tab).Id Full key prefix in Hex:
0x12-NNNNNNN-MMMMMMM

20. KeyType=00010011₂ FULL_DHT-CLASS_U_KEY
(Suffix: 298 bits) 362-bit Key
Db = {BFT-CLASS_U-DatabaseId}
+ Sch = default {Public-SchemaId (nulled)}
+ Tab = {BFT-CLASS_U-SchemaId}
+ Rec = {BFT-CLASS_U-TableId} Full key prefix in Hex:
0x13-NNNNNNN-MMMMMMM

21. KeyType=00010100₂ FULL_BFT-CLASS_UR_KEY
(Suffix: 448 bits) 512-bit Key (Database + Schema + Table + Record) =
(fullTableName.Record).Id =
Db.Sch.Tab.Record Full key prefix in Hex:
0x14-NNNNNNN-MMMMMMM

22. KeyType=00010101₂ FULL_DHT-CLASS_UR_KEY
(Suffix: 448 bits) 512-bit Key (Database + Schema + Table + Record) =
(fullTableName.Record).Id =
Db.Sch.Tab.Record Full key prefix in Hex:
0x15-NNNNNNN-MMMMMMM

In the Prefix portions to the left, “0xKK-NNNNNNN-MMMMMMM” stands for the 7-digit Enterprise Network Id in Hexadecimal Notation (NNNNNNN), (allowing for ~ 67 million networks), followed by 7 hex digits representing the Member Classes and/or Members in a Network (MMMMMMM) [where relevant and used – otherwise = 0000000]. The first 2 digits, KK, in the prefix (currently 0 – 22 in decimal order or 0 – 15 in hex), stand for the Key Types.

Thus, where there are 448 bit-systems (112 Hexadecimal digits), given in bits by 448 = 512 – 64 (64 = 8 + 28 + 28) [in Hex digits: 112 = 128 – 16 (16 = 2 + 7 + 7); in Bytes: 56 = 64 – 8 (8 = 1 + 3.5 + 3.5)], these enumerate all the Nodes, Databases, Schemas, Tables, Metadata Databases and Table types, Classes ‘A’ & ‘UR’ Records, Users and DHT “Values” for every Network and Member/Member-Class-within-Networks.

And, as there are 298 bits for Class U types, (75 Hexadecimal digits), given in bits by 298 = 512 – 214 (214 = 8 + 28 + 28 + 150 [so that the Class ‘U’ crossover schema fits]) [in Hex digits: 75 = 128 – 53 (53 = 2 + 7 + 7 + 37); in Bytes: 38 = 64 – 26 (26 = 1 + 3.5 + 3.5 + 18)], these enumerate all the Class U Databases, Schemas, Tables, DHT-(Metadata) Databases and Table types and DHT-Class ‘U’ Records for every Network and Member/Member-Class-within-Networks.

The minimum spanning spaces are given:

for Databases = 98 bits = to represent 3.17 x 10²⁹ databases

for Schemas = 100 bits = to represent 1.26 x 10³⁰ schemas

for Tables = 100 bits = to represent 1.26 x 10³⁰ tables

for Class ‘A’ Records = 150 bits = to represent 1.43 x 10⁴⁵ records

for Class ‘UR’ Records = 150 bits = to represent 1.43 x 10⁴⁵ records

for Class ‘U’ Records = 100 bits = to represent 1.26 x 10³⁰ records

John_L_Olsen

πάντα πιστός.

Always faithful.

In order to accomplish this, we recognise that the Class ‘U’ System must operate with a normal Prefix (64 bits) but a Suffix shortened by 150 to equal 298 bits. This is to enable the crossover public-schema-system, for Tx-Builder Metadata, to fit the structure we create, enabling the Id of the Client-facing BFT-System Class ‘U’ Table to give the Id needed for the corresponding DHT Class ‘U’ Record. The Class ‘U’ Record has the same suffix as the BFT Class-‘U’ Table (which is a Client-facing data form receiver). The Table Id is also mapped to the list of Use Cases for the Enterprise Network. The DHT Record Id gives access to the required Transaction-Builder Metadata, enabling appropriate construction of Prepared Statements for the Use Case Query under request.

The BFT- and DHT-Class-‘UR’ Systems represent keys whose suffixes correspond, thus tying together records in the arrays of Client-facing BFT Enterprise data-receiving tables with records storing the Transaction Traces of the former records. Investigations and Auditing are thereby enhanced.

High-Level Architecture Diagram: BFT-DHT
with jvm-libp2p Backbone

 +---------------------+       +-----------------------+
 |  Tertiary Nodes     | <---> | Secondary Nodes       |
 | (Client Apps)       |       | (Real-Time Ingest)    |
 +---------------------+       +-----------------------+
           |                             |
           v                             v
+----------------------+     +-----------------------------+
|  QUIC / gRPC         | <-> |  Supernode (Coordinator)     |
|  Transport Layer     |     |  - Controls block timing     |
|                      |     |  - Assigns blockId/slabId    |
+----------------------+     |  - Anchors Merkle Roots      |      
                             |  - Runs BFT-SMaRt Consensus  |                     
                              +-----------------------------+
                                          |
                                          v
                           +------------------------------+
                           | PostGIS Enterprise Database   |
                           | - Transaction Metadata        |
                           | - Slab / Block storage        |
                           +------------------------------+
                                          |
                                          v
                           +------------------------------+
                           | Ethereum / Elastos Anchor     |
                           | - Merkle Root Commitment      |
                           +------------------------------+

Additional:
  - libp2p handles peer discovery & routing (via Kademlia)
  - Each node has an IdType (512-bit SHA3-based ID)
  - Supernodes coordinate timing, traceability, and anchoring

~~~~~~~~~~~~~~~~~~~~~~~

Roadmap Checklist:
Immediate & Near-Term Milestones

Now (Completed or Near Completion)

Refactor around libp2p: QUIC, peer routing, NAT handling
Drop Carrier / etcd: Use Id + TransportContext only
Supernode Timer: Authority for blockHeight, slabId
GeneralClient QUIC Support: QUICChannel + failover routing
Metadata Pipeline: Collect 17 fields and store in PostGIS
MerkleTreeManager: Build leaves from sorted tx data

Next Steps (In-Progress)

Implement SlabWriter: Write metadata slab-by-slab to PostGIS
Implement SlabFinalizer: Trigger Merkle root calculation
Integrate EthereumAnchorClient: Anchor root every N txs
Connect blockId <--> traceId logic to Supernode
Resolve PeerId <--> Id using SHA3-derived rehash logic

After That (Nice to Have / Later Phases)

Create a Live Trace Visualizer (web-based UI)
Implement Multi-cluster anchor summaries
Add Peer-to-Peer Metrics Sync (cross-supernode collaboration)
Explore lightweight snapshotting for forensic exports

~~~~~~~~~~~~~~~~~~~~~~~

Id Generation Caller

Generates Sets of 512- and 362-bit Ids [fullTableName keys and Transaction Builder Metadata records] with 64-bit Prefixes. The clientIds [submitted as 44 x Base 58 characters] with full Prefixes [networkId = 0x0000000; submitted memberOrClass = 44 x Base 58] – all sha3-512 hashed, with output clientIds (and valueIds), producing full Prefixes and 448-bit Suffixes.


package bftsmart.demo.itotcca;

import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeoutException;

import bftsmart.demo.itotcca.IdManager.IdType;
import io.etcd.jetcd.ByteSequence;
import io.etcd.jetcd.Client;
import io.etcd.jetcd.kv.GetResponse;

public class IdGenerationCall {

	// Redis and etcd clients (assume these are initialized elsewhere in the application)
    
	private static final IdType FULL_CLASS_UR_KEY = null;
	private static final IdType FULL_CLASS_U_KEY = null;
	private static MemberOrClassId memberOrClassId;
	private static GeneralMessage request;
	private static final IdType FULL_CLASS_A_KEY = null;
	private static String recordOid;
	// Store the mapping in both Redis and Etcd
    public static void storeMapping(String key, String value) {
    	
        try {
            // Store in Etcd
            etcdClient.getKVClient().put(
                io.etcd.jetcd.ByteSequence.from(key.getBytes()),
                io.etcd.jetcd.ByteSequence.from(value.getBytes())
            );
        } catch (Exception e) {
            throw new RuntimeException("Error storing mapping: " + e.getMessage(), e);
        }
    }

 // Retrieve the mapping from Redis or Etcd
    
	private static Cluster networkId;
	private static final String MAPPING_PREFIX = "idMappings/";
	private static Client etcdClient = initializeEtcdClient();
	private static String databaseOid;
	private static String schemaOid;
	private static String tableOid;
	private static IdType idType;
	private static MemberOrClassId identifier;
	private static String supernodeIdStr;
	
	private static URI[] initializeEtcdEndpoints() {
	    String endpoints = System.getenv("ETCD_ENDPOINTS");

	    return Arrays.stream(endpoints.split(","))
	                 .map(String::trim)
	                 .map(URI::create)
	                 .toArray(URI[]::new);
	}


    // Method to initialize the Etcd client
    public static Client initializeEtcdClient() {
    	URI[] etcdEndpoints = initializeEtcdEndpoints();
        try {
            // Build and return the Etcd client
            return Client.builder()
                         .endpoints(etcdEndpoints)
                         .build();
        } catch (Exception e) {
            throw new IllegalStateException("Failed to initialize Etcd client: " + e.getMessage(), e);
        }
    }

    public static String getMapping(String key) {
        if (key == null || key.isEmpty()) {
            throw new IllegalArgumentException("Key cannot be null or empty");
        }

        try {
            ByteSequence keyBS = ByteSequence.from(MAPPING_PREFIX + key, StandardCharsets.UTF_8);
            CompletableFuture getFuture = etcdClient.getKVClient().get(keyBS);
            GetResponse response = getFuture.get(); // Blocking call to fetch value

            if (!response.getKvs().isEmpty()) {
                return response.getKvs().get(0).getValue().toString(StandardCharsets.UTF_8);
            }
            return null; // Key not found
        } catch (Exception e) {
            System.err.println("Error accessing Etcd: " + e.getMessage());
            e.printStackTrace();
            return null;
        }
    }

    
 // Generate or retrieve an ID for an entire Class_A Object
    public static String generateFullClassAKey(IdType FULL_CLASS_A_KEY, Cluster networkId, MemberOrClassId memberOrClassId, GeneralMessage request) throws Exception {
    	// Key includes IdType to enforce independence or consistency as needed
        String key = "FULL_CLASS_A_KEY-" + networkId + "-" + memberOrClassId + "-" + Long.toHexString(IdManager.oidMap.get("databaseOid")) + "-" + Long.toHexString(IdManager.oidMap.get("tableOid")) + "-" + Long.toHexString(IdManager.oidMap.get("tableOid")) + "-" + recordOid;

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createNewFullAId(IdType.FULL_CLASS_A_KEY, networkId, memberOrClassId, request);
        storeMapping(key, newId);
        return newId;
    }
    
 // Generate or retrieve an ID for an entire Class_UR Object
    public static String generateFullBFTClassURKey(Cluster networkId, MemberOrClassId memberOrClassId2, String databaseOid, String schemaOid, String tableOid, String recordOid) throws Exception {
    	// Key includes IdType to enforce independence or consistency as needed
        String key = "FULL_BFT_CLASS_UR_KEY-" + networkId + "-" + memberOrClassId2 + "-" + Long.toHexString(IdManager.oidMap.get("databaseOid")) + "-" + Long.toHexString(IdManager.oidMap.get("tableOid")) + "-" + Long.toHexString(IdManager.oidMap.get("tableOid")) + "-" + recordOid;

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createNewFullURId(IdType.FULL_BFT_CLASS_UR_KEY, networkId, memberOrClassId2, request);
        storeMapping(key, newId);
        return newId;
    }
    
 // Generate or retrieve an ID for an entire Class_U Object
    public static String generateFullBFTClassUKey(Cluster networkId, MemberOrClassId memberOrClassId, String databaseOid, String schemaOid, String tableOid) throws Exception {
    	// Key includes IdType to enforce independence or consistency as needed
        String key = "FULL_BFT_CLASS_U_KEY-" + networkId + "-" + memberOrClassId + "-" + Long.toHexString(IdManager.oidMap.get("databaseOid")) + "-" + Long.toHexString(2200L) + "-" + Long.toHexString(IdManager.oidMap.get("schemaOid")) + "-" + Long.toHexString(IdManager.oidMap.get("tableOid"));

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createAndStoreFullUId(IdType.FULL_BFT_CLASS_U_KEY, networkId, memberOrClassId, request);
        storeMapping(key, newId);
        return newId;
    }
    
 // Generate or retrieve an ID for an entire Class_UR Object
    public static String generateFullDHTClassURKey(Cluster networkId, MemberOrClassId memberOrClassId2, String databaseOid, String schemaOid, String tableOid, String recordOid) throws Exception {
    	// Key includes IdType to enforce independence or consistency as needed
        String key = "FULL_DHT_CLASS_UR_KEY-" + networkId + "-" + memberOrClassId2 + "-" + Long.toHexString(IdManager.oidMap.get("databaseOid")) + "-" + Long.toHexString(IdManager.oidMap.get("tableOid")) + "-" + Long.toHexString(IdManager.oidMap.get("tableOid")) + "-" + recordOid;

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createNewFullURId(IdType.FULL_DHT_CLASS_UR_KEY, networkId, memberOrClassId2, request);
        storeMapping(key, newId);
        return newId;
    }
    
 // Generate or retrieve an ID for an entire Class_U Object
    public static String generateFullDHTClassUKey(Cluster networkId, MemberOrClassId memberOrClassId, String databaseOid, String schemaOid, String tableOid) throws Exception {
    	// Key includes IdType to enforce independence or consistency as needed
        String key = "FULL_DHT_CLASS_U_KEY-" + networkId + "-" + memberOrClassId + "-" + Long.toHexString(IdManager.oidMap.get("databaseOid")) + "-" + Long.toHexString(2200L) + "-" + Long.toHexString(IdManager.oidMap.get("schemaOid")) + "-" + Long.toHexString(IdManager.oidMap.get("tableOid"));

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createAndStoreFullUId(IdType.FULL_DHT_TX_BLDER_META, networkId, memberOrClassId, request);
        storeMapping(key, newId);
        return newId;
    }
    
    /**public static String getSupernodeId(String networkId, MemberOrClassId identifier) {
        System.out.println("Beginning to getSupernodeId()");

        // Input validation
        if (networkId == null || networkId.isEmpty()) {
            throw new IllegalArgumentException("networkId cannot be null or empty");
        }
        
        System.out.println("Identifier passed: " + identifier);
        
        if (identifier == null) {
            throw new IllegalArgumentException("identifier cannot be null or empty");
        }

        // Construct the key
        String delimiter = "-";
        if (networkId.contains(delimiter)) {
            throw new IllegalArgumentException("Inputs should not contain the delimiter: " + delimiter);
        }
        String key = "PEER_NODE_SUPERNODE" + delimiter + networkId + delimiter + identifier;

        try {
            System.out.println("About to determine if SupernodeId() already exists.");
            System.out.flush();

            // Check if ID already exists
            String existingId = getMapping(key);
            if (existingId != null) {
                System.out.println("SupernodeId() already exists!");
                System.out.flush();
                return existingId;
            }

            System.out.println("About to create NEW SupernodeId().");
            System.out.flush();

            // Generate and store a new ID
            String newId = IdManager.createNewIdNonDb(
            	idType, 
            	networkId, 
            	identifier
            );

            //storeMapping(key, newId);
            System.out.printf("Generated and stored new supernode ID = %s%n", newId);
            System.out.flush();
            return newId;

        } catch (Exception e) {
            System.err.println("An exception occurred: " + e.getMessage());
            e.printStackTrace();
            // Depending on your application's requirements, you might want to rethrow the exception
            // or return a specific error value. Here, we'll return null to indicate failure.
            return null;
        }
    }**/


    
    // Generate or retrieve an ID for a secondary node
    public static String getSecondaryNodeId(IdType idType, Cluster networkId2, MemberOrClassId identifier) {
        System.out.println("Beginning to getSecondaryNodeId()");

        // Input validation
        if (networkId2 == null) {
            throw new IllegalArgumentException("networkId cannot be null or empty");
        }
        if (identifier == null) {
            throw new IllegalArgumentException("identifier cannot be null or empty");
        }

        // Construct the key
        String delimiter = "-";
        
        String key = "01" + delimiter + IdManager.getNetworkIdFromEnv().toString() + delimiter + IdManager.getMemberOrClassIdFromEnv().toString();

        try {
            System.out.println("About to determine if SecondaryNodeId() already exists.");
            System.out.flush();

            // Check if ID already exists
            String existingId = getMapping(key);
            if (existingId != null) {
                System.out.println("SecondaryNodeId() already exists!");
                System.out.flush();
                return existingId;
            }

            System.out.println("About to create NEW SecondaryNodeId().");
            System.out.flush();

            // Generate and store a new ID
            String newId = IdManager.createNewIdNonDb(
                idType,
                networkId2,
                memberOrClassId
            );

            storeMapping(key, newId);
            System.out.printf("Generated and stored new secondary node ID = %s%n", newId);
            System.out.flush();
            return newId;

        } catch (Exception e) {
            System.err.println("An exception occurred: " + e.getMessage());
            e.printStackTrace();
            // Depending on your application's requirements, you might want to rethrow the exception
            // or return a specific error value. Here, we'll return null to indicate failure.
            return null;
        }
    }

    
 // Generate or retrieve an ID for a tertiary node
    public static String getTertiaryNodeId(IdType idType, Cluster networkId, MemberOrClassId identifier) {
        System.out.println("Beginning to getSupernodeId()");

        // Input validation
        if (networkId == null) {
            throw new IllegalArgumentException("networkId cannot be null or empty");
        }
        if (identifier == null) {
            throw new IllegalArgumentException("identifier cannot be null or empty");
        }

        // Construct the key
        String delimiter = "-";
        
        String key = "02" + delimiter + networkId + delimiter + identifier;

        try {
            System.out.println("About to determine if TertiaryNodeId() already exists.");
            System.out.flush();

            // Check if ID already exists
            String existingId = getMapping(key);
            if (existingId != null) {
                System.out.println("TertiaryNodeId() already exists!");
                System.out.flush();
                return existingId;
            }

            System.out.println("About to create NEW tertiary nodeId().");
            System.out.flush();

            // Generate and store a new ID
            String newId = IdManager.createNewIdNonDb(
                idType,
                networkId,
                memberOrClassId
            );

            storeMapping(key, newId);
            System.out.printf("Generated and stored new tertiary node ID = %s%n", newId);
            System.out.flush();
            return newId;

        } catch (Exception e) {
            System.err.println("An exception occurred: " + e.getMessage());
            e.printStackTrace();
            // Depending on your application's requirements, you might want to rethrow the exception
            // or return a specific error value. Here, we'll return null to indicate failure.
            return null;
        }
    }

    
 // Generate or retrieve an ID for DhtTxBuilderMetadata
    public static String getDhtTxBuilderMetadataId(String networkId, MemberOrClassId memberOrClassId) throws NoSuchAlgorithmException, InterruptedException, ExecutionException, TimeoutException {
        // Use a common key format for databases, independent of Class
        String key = "DHT_TX_BLDER_META-" + networkId + "-" + memberOrClassId;

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createNewId(IdType.FULL_DHT_TX_BLDER_META, networkId, memberOrClassId, databaseOid, schemaOid, tableOid, recordOid);
        storeMapping(key, newId);
        return newId;
    }
    
 // Generate or retrieve an ID for a DhtTxTraceMetadata
    public static String getDhtTxTraceMetadataId(String networkId, MemberOrClassId memberOrClassId) throws NoSuchAlgorithmException, InterruptedException, ExecutionException, TimeoutException {
        // Use a common key format for databases, independent of Class
        String key = "DHT_TX_TRACE_META-" + networkId + "-" + memberOrClassId;

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createNewId(IdType.FULL_DHT_TX_BLDER_META, networkId, memberOrClassId, databaseOid, schemaOid, tableOid, recordOid);
        storeMapping(key, newId);
        return newId;
    }
    
 // Generate or retrieve an ID for a Role
    public static String getRoleId(String networkId, MemberOrClassId memberOrClassId) throws NoSuchAlgorithmException, InterruptedException, ExecutionException, TimeoutException {
        
        String key = "ROLE-" + networkId + "-" + memberOrClassId;

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createNewId(IdType.ROLE, networkId, memberOrClassId, databaseOid, schemaOid, tableOid, recordOid);
        storeMapping(key, newId);
        return newId;
    }
    
 // Generate or retrieve an ID for a Client
    public static String getClientId(Cluster networkId, MemberOrClassId base58PubKey) throws NoSuchAlgorithmException, InterruptedException, ExecutionException, TimeoutException {
        
        String key = "CLIENT_PERSONAL-" + "0000000" + "-" + base58PubKey;

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createNewIdNonDb(idType, networkId, memberOrClassId);
        storeMapping(key, newId);
        return newId;
    }
    
 // Generate or retrieve an ID for a Client
    public static String getDhtValueId(Cluster networkId, MemberOrClassId memberOrClassId) throws NoSuchAlgorithmException, InterruptedException, ExecutionException, TimeoutException {
        // Use a common key format for databases, independent of Class
        String key = "DHT_VALUE-" + networkId + "-" + memberOrClassId;

        // Check if ID already exists
        String existingId = getMapping(key);
        if (existingId != null) {
            return existingId;
        }

        // Generate and store a new ID
        String newId = IdManager.createNewIdNonDb(IdType.DHT_VALUE, networkId, memberOrClassId);
        storeMapping(key, newId);
        return newId;
    }

    public static void main(String[] args) {
        try {
            // Example: Generate a database ID
            String supernodeId = getSupernodeId(idType, networkId, memberOrClassId);
            System.out.println("Database ID: " + supernodeId);

            // Example: Generate a schema ID
            String secondaryNodeId = getSecondaryNodeId(idType, networkId, memberOrClassId);
            System.out.println("Schema ID: " + secondaryNodeId);

            // Example: Generate a table ID for Class U
            String tertiaryNodeId = getTertiaryNodeId(idType, networkId, memberOrClassId);
            System.out.println("Table ID (Class U): " + tertiaryNodeId);

            // Example: Generate a table ID for Class A
            String typeAKey = IdManager.createNewFullAId(FULL_CLASS_A_KEY, networkId, memberOrClassId, request);
            System.out.println("Full ID (Class A): " + typeAKey);

            // Example: Generate a record ID for Class UR
            String typeURKey = IdManager.createNewFullURId(FULL_CLASS_UR_KEY, networkId, memberOrClassId, request);
            System.out.println("Full ID (Class UR): " + typeURKey);
            
         // Example: Generate a table ID for Class A
            String typeUKey = IdManager.createAndStoreFullUId(FULL_CLASS_U_KEY, networkId, memberOrClassId, request);
            System.out.println("Table ID (Class U): " + typeUKey);

        } catch (Exception e) {
            System.err.println("Error: " + e.getMessage());
        }
    }


    public static String getSupernodeId(IdType idType, Cluster networkId2, MemberOrClassId supernodeClass) {
        System.out.println("Beginning to getSupernodeId()");

        // Input validation
        if (idType == null) {
            throw new IllegalArgumentException("idType cannot be null");
        }

        if (networkId2 == null) {
            throw new IllegalArgumentException("networkId cannot be null or empty");
        }

        if (supernodeClass == null) {
            throw new IllegalArgumentException("supernodeClass (identifier) cannot be null");
        }

        System.out.println("Identifier passed: " + supernodeClass); // ✅ Use supernodeClass directly

        // Construct the key
        String delimiter = "-";
        
        String networkHex = IdManager.getNetworkIdFromEnv().getId7(); // 👈 This gives you the 7-char hex
        String key = "00" + delimiter + IdManager.getNetworkIdFromEnv().getId7() + delimiter + supernodeClass.getIdentifier();


        try {
            System.out.println("About to determine if SupernodeId() already exists.");
            System.out.flush();

            // Check if ID already exists
            String existingId = getMapping(key);
            if (existingId != null) {
                System.out.println("SupernodeId() already exists!");
                System.out.flush();
                return existingId;
            }

            System.out.println("About to create NEW SupernodeId().");
            System.out.flush();

            // Generate and store a new ID
            String newId = IdManager.createNewIdNonDb(
                idType,
                networkId2,
                supernodeClass
            );

            //storeMapping(key, newId);
            System.out.printf("Generated and stored new supernode ID = %s%n", newId);
            System.out.printf("Generated supernodeIdStr = %s, networkId = %s, supernodeClass (memberOrClassId) = %s%n",
                    newId, IdManager.getNetworkIdFromEnv().getId7(), supernodeClass.getIdentifier());

            System.out.flush();
            return newId;

        } catch (Exception e) {
            System.err.println("An exception occurred in getSupernodeId(): " + e.getMessage());
            e.printStackTrace();
            return null;
        }
    }

}

Id Generation Class

Generates Sets of 512-, 462- [node and record keys] and 362-bit Ids [table keys and Transaction Builder Metadata records] with 64-bit Prefixes, or the clientIds [44 x Base 58 characters] with full Prefixes [networkId = 0x00; memberOrClass = 44 x Base 58] – all sha3-512 hashed, and valueIds [full Prefixes and 448-bit Suffixes)].


package bftsmart.demo.itotcca;

import java.math.BigInteger;
import java.net.URI;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.security.KeyFactory;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.spec.X509EncodedKeySpec;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.Arrays;
import java.util.HexFormat;
import java.util.Map;
import java.util.TreeMap;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeoutException;

import io.etcd.jetcd.ByteSequence;
import io.etcd.jetcd.Client;
import io.etcd.jetcd.KV;
import io.etcd.jetcd.kv.GetResponse;
import io.etcd.jetcd.options.GetOption;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import jakarta.xml.bind.DatatypeConverter;

import org.bouncycastle.jcajce.provider.digest.SHA3;
import org.elastos.did.crypto.Base58;
import bftsmart.demo.itotcca.CryptoBox.PublicKey;

public class IdManager {

    // bftsmart.demo.itotcca.IdManager.Id object representation
    public static class Id implements Comparable {
    	protected static final int SIZE = 512;
        public static final int BYTES = SIZE / Byte.SIZE; // Should be 64 (512/8)
        public static final int ID_BYTE_LENGTH = 64;

        public static Id MAX_ID;
        public static Id MIN_ID;
        
        /**static {
            try {
                System.out.println("[DEBUG] Initializing MAX_ID with IdType.DHT_VALUE...");
                byte[] maxIdBytes = new byte[BYTES];
                Arrays.fill(maxIdBytes, (byte) 0xFF);
                MAX_ID = new Id(maxIdBytes, IdType.DHT_VALUE);

                MIN_ID = new Id(new byte[BYTES], IdType.DHT_VALUE);
                System.out.println("[DEBUG] MAX_ID and MIN_ID initialized successfully.");
            } catch (Exception e) {
                System.err.println("[ERROR] Failed to initialize MAX_ID or MIN_ID: " + e.getMessage());
                throw new RuntimeException("Failed to initialize MAX_ID or MIN_ID", e);
            }
        }

        static {
            try {
                byte[] maxIdBytes = new byte[BYTES];
                Arrays.fill(maxIdBytes, (byte) 0xFF);
                MAX_ID = new Id(maxIdBytes, IdType.DHT_VALUE);

                MIN_ID = new Id(new byte[BYTES], IdType.DHT_VALUE);
            } catch (Exception e) {
                throw new RuntimeException("Failed to initialize MAX_ID or MIN_ID", e);
            }
        }**/
        


        protected IdType type;
        protected byte[] value;
        protected static byte[] bytes; // This holds the 64-byte (512-bit) ID
        protected int length;
        private String nodeId;
        private long lastSeen;
        private IdManager.Id id;
        
        public Id(byte[] idBytes, IdType type) {
            if (idBytes == null || idBytes.length != 64) {
                throw new IllegalArgumentException("ID must be 64 bytes long (SHA3-512)");
            }
            if (type == null) {
                throw new IllegalArgumentException("IdType cannot be null.");
            }
            this.bytes = Arrays.copyOf(idBytes, idBytes.length);
            this.type = type;
        }

        public Id(String string) {
            if (string == null) {
                throw new IllegalArgumentException("Prefix cannot be null.");
            }

            // Assume the Prefix class provides a method to convert itself to a byte array or ID representation.
            byte[] prefixBytes = string.getBytes();
            if (prefixBytes.length != BYTES) { // Must check BYTES, not SIZE
                throw new IllegalArgumentException("Prefix bytes must match the required ID byte size (64 bytes).");
            }

            this.bytes = Arrays.copyOf(prefixBytes, BYTES);
            this.type = null;
            this.value = new byte[0];
        }

     // Calculate XOR distance between two IDs
        public static byte[] distance(byte[] id2, byte[] bs2) {
            if (id2 == null || bs2 == null) {
                throw new IllegalArgumentException("Ids cannot be null.");
            }

            byte[] bytes1 = id2;  // ✅ Convert Id to byte array
            byte[] bytes2 = bs2;

            if (bytes1.length != bytes2.length) {
                throw new IllegalArgumentException("IDs must have the same length.");
            }

            byte[] resultBytes = new byte[bytes1.length];

            for (int i = 0; i < bytes1.length; i++) {
                resultBytes[i] = (byte) (bytes1[i] ^ bytes2[i]);  // ✅ XOR each byte
            }

            return resultBytes;  // ✅ Create a new Id with XOR result
        }



        // Generate a zero ID (all bytes set to 0)
        public static Id zero() {
            return new Id(new byte[BYTES], IdType.DHT_VALUE);
        }

		public static byte[] fromBase58(String base58) {
			return base58.getBytes();
		}

		// ✅ Fix getBytes() to ensure immutability
        public byte[] getBytes() {
            return Arrays.copyOf(bytes, bytes.length);
        }

        // ✅ Correct compareTo method
        public int compareTo(Id other) {
            if (other == null) {
                System.err.println("compareTo: other Id is NULL!");
                return 1; // Assume "this" is greater if comparing with null
                
            }
            if (other.bytes == null) {
                System.err.println("compareTo: other Id bytes are NULL!");
                return 1; // Assume "this" is greater if comparing with null
                
            }
            return Arrays.compareUnsigned(this.bytes, other.bytes);
        }


        // ✅ Fix toHexString() method
        public String toHexString() {
            StringBuilder hexString = new StringBuilder();
            for (byte b : bytes) {
                hexString.append(String.format("%02x", b));
            }
            return hexString.toString();
        }
        
    	public IdType getType() {
            return type;
        }
    	
        public byte[] getValue() {
            return value;
        }

        
        public boolean matchesType(IdType expectedType) {
            return type == expectedType;
        }

        public static Id ofHex(String supernodeNodeId) {
            if (supernodeNodeId.length() != 128) {
                throw new IllegalArgumentException("Hex string must be 128 characters long to represent a 64-byte ID.");
            }
            byte[] decodedBytes = (supernodeNodeId).getBytes();
            try {
				return new Id(bytes, IdType.DHT_VALUE);
			} catch (Exception e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}  // ✅ or another valid IdType
			return MAX_ID;
        }

        private static byte[] decodeHex(byte[] workerNodeId) {
            if (workerNodeId.length != 128) { // Validate length for a 64-byte ID (512 bits)
                throw new IllegalArgumentException("Hex string must be 128 characters long to represent a 64-byte ID.");
            }
            int length = workerNodeId.length;
            byte[] data = new byte[length / 2];
            for (int i = 0; i < length; i += 2) {
                data[i / 2] = (byte) ((Character.digit(workerNodeId.length, 16) << 4)
                                      + Character.digit(workerNodeId.length, 16));
            }
            return data;
        }


    	
    	
    	/**
    	 * sorts the closest entries to the head, the furth est to the tail
    	 */
    	public static final class Comparator implements java.util.Comparator {
    		private final Id target;

    		public Comparator(Id target) {
    			this.target = target;
    		}

    		public int compare(int o1, IdManager.Id o2) {
    			return (int) target.threeWayCompare(o1, o2);
    		}

			@Override
			public int compare(Id arg0, Id arg1) {
				// TODO Auto-generated method stub
				return 0;
			}
    	}

    	/**
    	 * Construct a random kademlia Id.
    	 */
    	protected Id() {
    		bytes = new byte[BYTES];
			this.idBytes = null;
    	}

    	public int threeWayCompare(int o1, Id o2) {
			// TODO Auto-generated method stub
			return 0;
		}

		/**
    	 * Construct a random kademlia Id.
    	 */
    	protected Id(Id id) {
    		new Id(bytes, IdType.DHT_VALUE); ;
    	}
    	
    	public static Id maxId(IdType type) {
    	    byte[] maxIdBytes = new byte[BYTES];
    	    Arrays.fill(maxIdBytes, (byte) 0xFF);
    	    return new Id(maxIdBytes, type);
    	}

    	public static Id minId(IdType type) {
    	    return new Id(new byte[BYTES], type);
    	}


    	/**
    	 * Construct the kademlia Id from a given binary id
    	 *
    	 * @param id the binary id in bytes
    	 */
    	protected Id(byte[] buf, int pos) {
    		this.bytes = Arrays.copyOfRange(buf, pos, pos + BYTES);
			this.idBytes = null;
    	}

    	public static Id of(byte[] buf, int pos) {
    		if (buf.length - pos < BYTES)
    			throw new IllegalArgumentException("Binary id should be " + BYTES + " bytes long.");

    		return new Id(buf, pos);
    	}

    	public static String of(String id) {
            if (id.startsWith("0x")) return Arrays.toString(id.getBytes());
            return ofBase58(id);
        }

    	/**
		 * Create a Carrier Id from base58 string.
		 *
		 * @param base58Id the id string
		 */
    	public static String ofBase58(String base58Id) {
    		return of(Base58.decode(base58Id));
    	}



    	public static Id random(IdType idType, bftsmart.demo.itotcca.Cluster networkId2, MemberOrClassId memberOrClassId) {
    	    if (networkId2 == null || memberOrClassId == null) {
    	        throw new IllegalArgumentException("Network ID and Member/Class ID cannot be null.");
    	    }

    	    // Generate 64-bit (8-byte) prefix
    	    byte[] prefixBytes = generateNonDbIdPrefix(idType, networkId2, memberOrClassId);

    	    // Generate 448-bit (56-byte) random portion
    	    byte[] randomBytes = new byte[56];
    	    SecureRandom secureRandom = new SecureRandom();  // Ensure fresh randomness
    	    secureRandom.nextBytes(randomBytes);  // Fill with truly random bytes

    	    // Concatenate prefix + random part
    	    byte[] fullIdBytes = new byte[64]; // 512 bits = 64 bytes
    	    System.arraycopy(prefixBytes, 0, fullIdBytes, 0, 8);   // Copy prefix (8 bytes)
    	    System.arraycopy(randomBytes, 0, fullIdBytes, 8, 56);  // Copy random part (56 bytes)

    	    return new Id(fullIdBytes, idType);
    	}


    	private static final int RECORD_ID_BITS = 150;
        private static final int RECORD_ID_BYTES = (int) Math.ceil(RECORD_ID_BITS / 8.0); // 19 bytes

        public static String randomRecordIdHex() {
            // Generate 19 random bytes
            byte[] randomBytes = new byte[RECORD_ID_BYTES];
            SecureRandom secureRandom = new SecureRandom(); // Cryptographically strong random source
            secureRandom.nextBytes(randomBytes); // Fill the byte array with secure random data

            // Convert to Hex String (uppercase for clarity)
            return HexFormat.of().formatHex(randomBytes).toUpperCase();
        }

        public static void main(String[] args) {
            // Test: Generate a random 150-bit hex string
            String randomHexId = randomRecordIdHex();
            System.out.println("Random 150-bit Record ID (Hex): " + randomHexId);
        }

    	public byte[] bytes() {
    		return bytes;
    	}

    	public int getInt(int offset) {
    		return Byte.toUnsignedInt(bytes[offset]) << 24 |
    				Byte.toUnsignedInt(bytes[offset+1]) << 16 |
    				Byte.toUnsignedInt(bytes[offset+2]) << 8 |
    				Byte.toUnsignedInt(bytes[offset+3]);
    	}

    	public Id add(Id id) {
    		Id result = new Id();

    		byte[] a = bytes;
    		byte[] b = id.bytes;
    		byte[] r = result.bytes;

    		int carry = 0;
    		for(int i = BYTES - 1; i >= 0; i--) {
    			carry = (a[i] & 0xff) + (b[i] & 0xff) + carry;
    			r[i] = (byte)(carry & 0xff);
    			carry >>>= 8;
    		}

    		return result;
    	}

    	/**
    	 * Checks the distance between this and another Id
    	 *
    	 * @param id another Id
    	 *
    	 * @return The distance of this NodeId from the given NodeId
    	 */
    	public byte[] distance(byte[] to) {
    		return distance(this, to);
    	}


    	private byte[] distance(Id id2, byte[] to) {
    	    if (id2 == null || to == null) {
    	        throw new IllegalArgumentException("IDs cannot be null.");
    	    }

    	    byte[] idBytes = id2.getBytes();  // Convert Id to byte array
    	    if (idBytes.length != to.length) {
    	        throw new IllegalArgumentException("Byte arrays must have the same length.");
    	    }

    	    byte[] result = new byte[idBytes.length];
    	    for (int i = 0; i < idBytes.length; i++) {
    	        result[i] = (byte) (idBytes[i] ^ to[i]);  // Perform XOR operation byte-by-byte
    	    }
    	    return result;
    	}

		/**
    	 * Get an Id that is some distance away from this Id
    	 *
    	 * @param distance in number of bits
    	 * @return the newly generated Id
    	 */
    	public byte[] getIdByDistance(int distance) {
    		byte[] result = new byte[BYTES];

    		int zeroBytes = (SIZE - distance) / 8;
    		int zeroBits = (SIZE - distance) % 8;

    		// new byte array is initialized with all zeroes
    		// Arrays.fill(result, 0, zeroBytes, (byte)0);

    		if (zeroBytes < BYTES) {
    			result[zeroBytes] = (byte)(0xFF >>> zeroBits);

    			Arrays.fill(result, zeroBytes + 1, BYTES, (byte) 0xFF);
    		}

    		return Id.of(result).getBytes();
    	}




    	/**
    	 * Counts the number of leading 0's in this Id
    	 *
    	 * @return the number of leading 0's
    	 */
    	public int getLeadingZeros() {
    		int msb = 0;

    		int i;
    		for (i = 0; i < BYTES && bytes[i] == 0; i++);
    		msb += i << 3;

    		if (i < BYTES) {
    			byte b = bytes[i];
    			if (b > 0) {
    				int n = 7;
    				if (b >= 1 <<  4) { n -=  4; b >>>=  4; }
    				if (b >= 1 <<  2) { n -=  2; b >>>=  2; }
    				msb += (n - (b >>> 1));
    			}
    		}

    		return msb;
    	}

    	/**
    	 * Counts the number of trailing 0's in this Id
    	 *
    	 * @return the number of trailing 0's
    	 */
    	public int getTrailingZeros() {
    		int lsb = 0;

    		int i;
    		for (i = BYTES - 1; i >= 0 && bytes[i] == 0; i--);
    		lsb += (BYTES - 1 - i) << 3;

    		if (i >= 0) {
    			byte b = (byte)(~bytes[i] & (bytes[i] - 1));
    			if (b <= 0) {
    				lsb += (b & 8);
    			} else {
    				if (b > 1 <<  4) { lsb +=  4; b >>>=  4; }
    				if (b > 1 <<  2) { lsb +=  2; b >>>=  2; }
    				lsb += ((b >>> 1) + 1);
    			}
    		}

    		return lsb;
    	}

    	/**
    	 * Check if the leading bits up to the Nth bit of both Id are equal
    	 *
    	 *     	 *   n = -1: no bits have to match
    	 *   n >= 0: n bytes have to match
    	 * 
    	 */
    	protected static boolean bitsEqual(Id id1, Id id2, int n) {
    		if (n < 0)
    			return true;

    		int mmi = Arrays.mismatch(id1.bytes, id2.bytes);

    		int indexToCheck = n >>> 3;
    		int diff = (id1.bytes[indexToCheck] ^ id2.bytes[indexToCheck]) & 0xff;

    		boolean bitsDiff = (diff & (0xff80 >>> (n & 0x07))) == 0;

    		return mmi == indexToCheck ? bitsDiff : Integer.compareUnsigned(mmi, indexToCheck) > 0;
    	}

    	protected static void bitsCopy(Id src, Id dest, int depth) {
    		if (depth < 0)
    			return;

    		// copy over all complete bytes
    		int idx = depth >>> 3;
    		if (idx > 0)
    			System.arraycopy(src.bytes, 0, dest.bytes, 0, idx);

    		int mask = 0xFF80 >>> (depth & 0x07);

    		// mask out the part we have to copy over from the last prefix byte
    		dest.bytes[idx] &= ~mask;
    		// copy the bits from the last byte
    		dest.bytes[idx] |= src.bytes[idx] & mask;
    	}

    	public Signature.PublicKey toSignatureKey() {
    		return Signature.PublicKey.fromBytes(bytes);
    	}


    	/**
    	 * Compares a NodeId to this NodeId
    	 *
    	 * @param o The NodeId to compare to this NodeId
    	 * @return boolean Whether the 2 NodeIds are equal
    	 */
    	@Override
    	public boolean equals(Object o) {
    		if (o == this)
    			return true;

    		if (o instanceof Id) {
    			Id id = (Id) o;
    			return Arrays.equals(this.bytes, id.bytes);
    		}
    		return false;
    	}

    	@Override
    	public int hashCode() {
    		byte[] b = bytes;

    		return (((b[0] ^ b[1] ^ b[2] ^ b[3] ^ b[4] ^ b[5] ^ b[6] ^ b[7]) & 0xff) << 24)
    				| (((b[8] ^ b[9] ^ b[10] ^ b[11] ^ b[12] ^ b[13] ^ b[14] ^ b[15]) & 0xff) << 16)
    				| (((b[16] ^ b[17] ^ b[18] ^ b[19] ^ b[20] ^ b[21] ^ b[22] ^ b[23]) & 0xff) << 8)
    				| ((b[24] ^ b[25] ^ b[26] ^ b[27] ^ b[28] ^ b[29] ^ b[30] ^ b[31]) & 0xff);
    	}

    	/**
    	 * @return The BigInteger representation of the key
    	 */
    	public BigInteger toInteger() {
    		return new BigInteger(1, bytes);
    	}

    	public String toBase58String() {
    		return Base58.encode(bytes);
    	}

    	public String toBinaryString() {
    		StringBuilder repr = new StringBuilder(SIZE + (SIZE >>> 2));

    		for(int i = 0; i < SIZE; i++) {
    			repr.append((bytes[i >>> 3] & (0x80 >> (i & 0x07))) != 0 ? '1' : '0');
    			if ((i & 0x03) == 0x03) repr.append(' ');
    		}
    		return repr.toString();
    	}

        
        
        
        public boolean isRecordId() {
            // Check if the ID's first 3 bits correspond to the ENTERPRISE_RECORD prefix "100"
            return (bytes[0] & 0xE0) == 0x80; // 0xE0 masks the first 3 bits, 0x80 is binary 10000000
        }

        public boolean isNodeId() {
            return (bytes[0] & 0xE0) == 0x00; // Check if the first 3 bits are 000
        }

        public boolean isValueId() {
            // Check if the first 3 bits of the first byte are "111" (binary)
            return (bytes[0] & 0xE0) == 0xE0;
        }

        public boolean isPeerId() {
            // Check if the first 3 bits of the first byte are "000" (binary)
            return (bytes[0] & 0xE0) == 0x00;
        }

    	public static bftsmart.demo.itotcca.Id of(bftsmart.demo.itotcca.Id id2) {
            if (id2 == null || id2.length != 64) {
                throw new IllegalArgumentException("Invalid byte array for Id.");
            }
            return (bftsmart.demo.itotcca.Id) id2;
        }
    	
    	public static bftsmart.demo.itotcca.Prefix of(Prefix prefix) {
            if (prefix == null || prefix.length != 64) {
                throw new IllegalArgumentException("Invalid byte array for Id.");
            }
            return (bftsmart.demo.itotcca.Prefix) prefix;
        }
        
        public boolean isEnterpriseRecord() {
            return (bytes[0] & 0xE0) == 0x80;
        }

        public boolean isTransactionTrace() {
            return (bytes[0] & 0xE0) == 0xA0;
        }

        public boolean isTableId() {
            return (bytes[0] & 0xE0) == 0x40;
        }

    	public static Id ofBit(int depth) {
    	    if (depth < 0 || depth >= SIZE * 8) {
    	        throw new IllegalArgumentException("Depth must be in the range [0, 511].");
    	    }

    	    byte[] idBytes = new byte[SIZE];
    	    int byteIndex = depth / 8;
    	    int bitIndex = 7 - (depth % 8);

    	    idBytes[byteIndex] = (byte) (1 << bitIndex);

    	    return new Id(idBytes, IdType.DHT_VALUE);
    	}
    	
    	    // Compare this Id with the first Id
    	    int compareToFirst = this.compareTo((IdManager.Id) id);
			

    	    
    	    // Compare this Id with the last Id
    	    int compareToLast = this.compareTo((IdManager.Id) id);

			public static byte[] hashValue(String input) {
				// TODO Auto-generated method stub
				return null;
				
    	}


    	public PublicKey toEncryptionKey() {
            try {
                X509EncodedKeySpec spec = new X509EncodedKeySpec(this.bytes);
                KeyFactory keyFactory = KeyFactory.getInstance("RSA");
                return (PublicKey) keyFactory.generatePublic(spec);
            } catch (Exception e) {
                throw new RuntimeException("Failed to convert Id to PublicKey: " + e.getMessage(), e);
            }
        }

    	public static void bitsCopy(Prefix source, Id firstId, int depth) {
    	    if (source == null || firstId == null) {
    	        throw new IllegalArgumentException("Source and destination prefixes cannot be null.");
    	    }

    	    byte[] sourceBytes = source.getBytes();
    	    byte[] destinationBytes = firstId.getBytes();

    	    System.arraycopy(sourceBytes, 0, destinationBytes, 0, depth / 8); // Copy full bytes

    	    int remainingBits = depth % 8;
    	    if (remainingBits > 0) {
    	        int mask = 0xFF00 >>> remainingBits; // Mask for remaining bits
    	        destinationBytes[depth / 8] &= ~mask; // Clear the remaining bits
    	        destinationBytes[depth / 8] |= sourceBytes[depth / 8] & mask; // Set bits from source
    	    }
    	}

		public Object threeWayCompare(byte[] id1, byte[] id2) {
			// TODO Auto-generated method stub
			return null;
		}

		public static bftsmart.demo.itotcca.IdManager.Id of(bftsmart.demo.itotcca.IdManager.Id id2) {
            if (id2 == null || id2.length != 64) {
                throw new IllegalArgumentException("Invalid byte array for Id.");
            }
            return (bftsmart.demo.itotcca.IdManager.Id) id2;
        }

		public static String of(byte[] keyBytes) {
		    if (keyBytes == null || keyBytes.length == 0) {
		        throw new IllegalArgumentException("Key bytes must not be null or empty.");
		    }

		    // Ensure the key size is correct (e.g., 64 bytes for SHA3-512 IDs)
		    if (keyBytes.length != SIZE) { // SIZE must be defined elsewhere
		        throw new IllegalArgumentException("Invalid key length. Expected " + SIZE + " bytes.");
		    }

		    // ✅ Simply return the original keyBytes (no need to wrap in Id object)
		    return Arrays.toString(keyBytes.clone()); // Returns a defensive copy to prevent modification
		}

		
		public String toHex() {
		    StringBuilder hexString = new StringBuilder();
		    byte[] idBytes = this.getBytes(); // Assuming `getBytes()` returns the ID as a byte array.

		    for (byte b : idBytes) {
		        hexString.append(String.format("%02x", b));
		    }

		    return hexString.toString();
		}

		public static Id from(bftsmart.demo.itotcca.Id sender) {
		    if (sender == null) {
		        throw new IllegalArgumentException("Sender ID cannot be null.");
		    }
		    // Assuming that the `Id` class can directly use the bytes from `sender`
		    byte[] senderBytes = sender.getBytes(); // Ensure the `Id` class has a `getBytes()` method
		    return new Id(senderBytes, IdType.DHT_VALUE);
		}
		
		public Id getId() {
		    return this; // Return the current Id instance
		}

		public boolean isValidForLookup() {
		    // Check if the node is reachable and active
		    if (!isOnline() || !isReachable()) {
		        return false;
		    }

		    // Additional criteria for validity can be added here
		    // Example: Ensure the node has a valid ID and is not marked as stale
		    return this.nodeId != null && !isStale();
		}

		public boolean isReachable() {
			// TODO Auto-generated method stub
			return false;
		}

		private boolean isOnline() {
			// TODO Auto-generated method stub
			return false;
		}

		// Example auxiliary methods that could support the logic
		private boolean isStale() {
		    // Assume we have a `lastSeen` timestamp and a threshold
		    long currentTime = BFTTimeService.getLeaderTime();
		    long staleThreshold = 5 * 60 * 1000; // 5 minutes
		    return (currentTime - this.lastSeen) > staleThreshold;
		}

		public static String toHexString(IdManager.Id id) {
		    if (id == null || id.length == 0) {
		        throw new IllegalArgumentException("Byte array cannot be null or empty.");		    
		    }
		    return toHexString(id);
		}
		
		public int calculateDistance(bftsmart.demo.itotcca.IdManager.Id nodeId) {
			// TODO Auto-generated method stub
			return 0;
		}


		// Factory method for special cases
	    public static Id emptyId() {
	        return new Id(new byte[0], IdType.DHT_VALUE);
	    }

	    private byte[] idBytes;

        public Id(Id maxIdBytes2, IdType dhtValue) {
			this.idBytes = null;
			// TODO Auto-generated constructor stub
		}

        public Id(IdType type2, String hexString) {
            if (type2 == null) {
                throw new IllegalArgumentException("IdType cannot be null.");
            }
            if (hexString == null || hexString.length() != 2 * BYTES) {
                throw new IllegalArgumentException("Hex string must be " + (2 * BYTES) + " characters long.");
            }

            this.bytes = decodeHex(hexString.getBytes());
            MAX_ID.type = type2;
        }

		/**
         * Converts the ID into a byte array representation.
         * @return Byte array of the ID.
         */
        public byte[] toByteArray() {
            return Arrays.copyOf(idBytes, idBytes.length);
        }

        /**
         * Converts the ID into a BigInteger for mathematical operations.
         * @return BigInteger representation of the ID.
         */
        public BigInteger toBigInteger() {
            return new BigInteger(1, idBytes);
        }

        public int getBitLength() {
            return idBytes.length * 8; // 64 bytes * 8 bits = 512 bits
        }
		
		public static bftsmart.demo.itotcca.KBucketEntry ofHexK(String nodeIdStr) {
            if (nodeIdStr.length() != 128) {
                throw new IllegalArgumentException("Hex string must be 128 characters long to represent a 64-byte ID.");
            }
            byte[] decodedBytes = decodeHex(nodeIdStr.getBytes());
            return new bftsmart.demo.itotcca.KBucketEntry(decodedBytes);
        }

		public static Id fromBytes(byte[] byteArray) {
		    if (byteArray == null || byteArray.length != 64) { // Assuming 512-bit (64-byte) Ids
		        throw new IllegalArgumentException("Invalid byte array length. Expected 64 bytes (512-bit Id).");
		    }
		    
		    try {
		        // Convert byte array to a hexadecimal string representation
		        String hexString = DatatypeConverter.printHexBinary(byteArray).toLowerCase();

		        // Create and return a new Id object
		        return new Id(IdType.DHT_VALUE, hexString);
		    } catch (Exception e) {
		        throw new RuntimeException("Error parsing byte array to Id", e);
		    }
		}

		public String approxDistance(Id target, Id target2) {
		    if (target == null || target2 == null) {
		        throw new IllegalArgumentException("Target IDs cannot be null.");
		    }

		    byte[] bytes1 = target.getBytes();
		    byte[] bytes2 = target2.getBytes();

		    int diffCount = 0;

		    // XOR each byte to count bit differences
		    for (int i = 0; i < bytes1.length; i++) {
		        diffCount += Integer.bitCount(bytes1[i] ^ bytes2[i]); // Count differing bits
		    }

		    return String.valueOf(diffCount); // Return as a string
		}

		public int compareTo(Prefix prefix) {
			// TODO Auto-generated method stub
			return 0;
		}

		public int threeWayCompare(Id id1, Id id2) {
			// TODO Auto-generated method stub
			return 0;
		}

		public int threeWayCompare(byte[] lastKey, Id neighborInfo) {
			// TODO Auto-generated method stub
			return 0;
		}

		public Id distance(Id maxId, Id target) {
			// TODO Auto-generated method stub
			return null;
		}

		public byte[] getAddress() {
			// TODO Auto-generated method stub
			return null;
		}

		
    }
   
    private static final String ETCD_ENDPOINTS = System.getenv("ETCD_ENDPOINTS");
    private static Client etcdClient = initializeEtcdClient();

	private static IdType idType;
	public static Map oidMap;
	private static String databaseOid;
	private static String schemaOid;
	private static String tableOid;
	private static String fullTableName;
	private static String recordOid;
	private static String recordId;
	private static byte[] suffixBytes;
	private static Object result;
	private static String containerName;
	private static String txId;
	private static int entnum;
	private static long slabId;
	private static String networkId;
	private static String databaseId;
	private static String schemaId;
	private static String tableId;
	private static String blockId;
	private static String value;
	
	private static URI[] initializeEtcdEndpoints() {
	    String endpoints = ETCD_ENDPOINTS;

	    return Arrays.stream(endpoints.split(","))
	                 .map(String::trim)
	                 .map(URI::create)
	                 .toArray(URI[]::new);
	}


    // Method to initialize the Etcd client
    public static Client initializeEtcdClient() {
    	URI[] etcdEndpoints = initializeEtcdEndpoints();
        try {
            // Build and return the Etcd client
            return Client.builder()
                         .endpoints(etcdEndpoints)
                         .build();
        } catch (Exception e) {
            throw new IllegalStateException("Failed to initialize Etcd client: " + e.getMessage(), e);
        }
    }
    
    
    private static final String clusterName = System.getenv("CLUSTER");

	public static bftsmart.demo.itotcca.Cluster parseCluster(String clusterName) {
	    if (clusterName == null || clusterName.isBlank()) {
	        throw new IllegalArgumentException("❌ Environment variable 'CLUSTER' is missing or empty!");
	    }

	    try {
	        // Convert environment variable to uppercase for Enum lookup
	        return bftsmart.demo.itotcca.Cluster.valueOf(clusterName.toUpperCase());
	    } catch (IllegalArgumentException e) {
	        throw new IllegalArgumentException("❌ Invalid cluster name: " + clusterName, e);
	    }
	}
	
	// If you need lowercase for internal use:
	public static String getClusterNameLowerCase() {
	    return parseCluster(clusterName).name().toLowerCase(); // ✅ Now safely converts enum name to lowercase
	}
    
    
	public static void mapTableOidToUseCase(String networkId, String database, String schemaName, String tableName) throws Exception {
	    // Retrieve OIDs from Redis cache or PostgreSQL
	    Map oidMap = getOIDs(networkId, database, schemaName, tableName);

	    if (!oidMap.containsKey("databaseOid") || !oidMap.containsKey("schemaOid") || !oidMap.containsKey("tableOid")) {
	        throw new RuntimeException("Failed to retrieve necessary OIDs for mapping to use case.");
	    }

	    // Extract OIDs
	    long databaseOid = oidMap.get("databaseOid");
	    long schemaOid = oidMap.get("schemaOid");
	    long tableOid = oidMap.get("tableOid");

	    // Construct Use Case mapping keys
	    String keyDatabase = "dbToUseCase:" + databaseOid;
	    String keySchema = "schemaToUseCase:" + schemaOid;
	    String keyTable = "tableToUseCase:" + tableOid;

	    // Convert OIDs to byte[]
	    byte[] databaseOidBytes = ByteBuffer.allocate(Long.BYTES).putLong(databaseOid).array();
	    byte[] schemaOidBytes = ByteBuffer.allocate(Long.BYTES).putLong(schemaOid).array();
	    byte[] tableOidBytes = ByteBuffer.allocate(Long.BYTES).putLong(tableOid).array();

	    // Store OIDs in etcd and Redis
	    putToEtcd(keyDatabase, databaseOidBytes);
	    
	    putToEtcd(keySchema, schemaOidBytes);
	    
	    putToEtcd(keyTable, tableOidBytes);
	    
	}


    /**
     * Retrieves both the database OID and the schema.table OID.
     *
     * @param networkId The network identifier.
     * @param database  The database name.
     * @param schemaName The schema name.
     * @param tableName The table name.
     * @return A map containing the database OID and table OID.
     * @throws Exception If a database access error occurs.
     */
	public static Map getOIDs(String networkId, String database, String schemaName, String tableName) throws Exception {
	    Map oidMap = new TreeMap<>();

	    // If all OIDs are cached, return them
	    if (oidMap.containsKey("databaseOid") && oidMap.containsKey("schemaOid") && oidMap.containsKey("tableOid")) {
	        return oidMap;
	    }

	    String query = "SELECT d.oid AS database_oid, n.oid AS schema_oid, c.oid AS table_oid " +
	                   "FROM pg_database d " +
	                   "JOIN pg_namespace n ON n.nspname = ? " +
	                   "JOIN pg_class c ON c.relnamespace = n.oid AND c.relname = ? " +
	                   "WHERE d.datname = ?";

	    int replicaId = Integer.valueOf(System.getenv("NODE_ID"));
        Connection[] connArr = DatabaseConnector.getConnection(parseCluster(clusterName));
        try (Connection conn = connArr[replicaId];
	         PreparedStatement statement = conn.prepareStatement(query)) {

	        statement.setString(1, schemaName);
	        statement.setString(2, tableName);
	        statement.setString(3, database);

	        try (ResultSet resultSet = statement.executeQuery()) {
	            if (resultSet.next()) {
	                long dbOid = resultSet.getLong("database_oid");
	                long schemaOid = resultSet.getLong("schema_oid");
	                long tableOid = resultSet.getLong("table_oid");

	                oidMap.put("databaseOid", dbOid);
	                oidMap.put("schemaOid", schemaOid);
	                oidMap.put("tableOid", tableOid);

	                
	            }
	        }
	    } catch (SQLException e) {
	        throw new RuntimeException("Failed to retrieve database, schema, and table OIDs from PostgreSQL", e);
	    }

	    return oidMap;
	}


	public static Id generateTransactionTraceId(String networkId, MemberOrClassId memberOrClassId, String database, String schema, String table, String recordId, UUID txId) {
	    // Validate inputs
	    if (recordId == null || recordId.isEmpty()) {
	        throw new IllegalArgumentException("Record ID must not be null or empty.");
	    }

	    // Retrieve the suffix stored in Redis/etcd
	    String cacheKey = "DHT_TX_TRACE_SUFFIX:" + database + "." + schema + "." + table + "." + recordId;
	    
	    suffixBytes = getFromEtcd(cacheKey);
	    

	    String suffix = (suffixBytes != null) ? new String(suffixBytes, StandardCharsets.UTF_8) : "DEFAULT_SUFFIX";

	    // Create a ByteBuffer with the required ID format
	    ByteBuffer buffer = ByteBuffer.allocate(96); // Ensure enough space for all fields
	    buffer.put((byte) 0b10100000);
	    buffer.putLong(Long.parseLong(networkId, 16)); // Convert hex networkId to long
	    buffer.putLong(Long.parseLong(memberOrClassId.toString(), 16));
	    buffer.putLong(Long.parseLong(database, 16));
	    buffer.putLong(Long.parseLong(schema, 16));
	    buffer.putLong(Long.parseLong(table, 16));
	    buffer.putLong(Long.parseLong(recordId, 16));
	    buffer.putLong(txId.getMostSignificantBits());
	    buffer.putLong(txId.getLeastSignificantBits());

	    // Append the suffix
	    byte[] suffixData = suffix.getBytes(StandardCharsets.UTF_8);
	    buffer.put(suffixData, 0, Math.min(suffixData.length, buffer.remaining()));

	    return new Id(buffer.array(), IdType.FULL_DHT_TX_BLDER_META);
	}


	public static Id getUseCaseMetadataId(GeneralMessage request, IdType idType) {
	    String key = "tableToUseCase:" + request.getFullTableName();
	    result = getFromEtcd(key);
	    if (result != null) {
	        String suffix = new String();
	        String useCaseId = "DHT_TX_BLDER_META-" + suffix;
	        return new Id(useCaseId.getBytes(StandardCharsets.UTF_8), IdType.FULL_DHT_TX_BLDER_META);
	    }

	    return null;
	}


    public static void putToEtcd(String key, byte[] value) {
        try {
            etcdClient.getKVClient().put(ByteSequence.from(key.getBytes()), ByteSequence.from(value)).get();
        } catch (Exception e) {
            throw new RuntimeException("Failed to put data to etcd", e);
        }
    }

    private static byte[] getFromEtcd(String key) {
        try {
            GetResponse response = etcdClient.getKVClient().get(ByteSequence.from(key.getBytes()), GetOption.DEFAULT).get();
            if (!response.getKvs().isEmpty()) {
                return response.getKvs().get(0).getValue().getBytes();
            }
        } catch (Exception e) {
            throw new RuntimeException("Failed to get data from etcd", e);
        }
        return null;
    }

    // Generate an ID
    public static bftsmart.demo.itotcca.IdManager.Id generateId(IdType type, String input) {
        try {
            MessageDigest digest = MessageDigest.getInstance("SHA3-512");
            byte[] hash = digest.digest(input.getBytes());
            String binaryHash = bytesToBinaryString(hash);
            return new bftsmart.demo.itotcca.IdManager.Id(type, binaryHash.substring(3)); // Add prefix
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException("SHA3-512 algorithm not found", e);
        }
    }
    
    public static Id generateId(IdType type, String networkId, MemberOrClassId memberOrClassId, String databaseOid, String schemaOid, String tableOid, String recordOid) {
        try {
            MessageDigest sha3 = MessageDigest.getInstance("SHA3-512");
            String fullKey = type.getHexPrefix() + networkId + memberOrClassId + databaseOid + schemaOid + tableOid;
            if (recordOid != null) fullKey += recordOid;
            byte[] hashedId = sha3.digest(fullKey.getBytes(StandardCharsets.UTF_8));
            return new Id(Arrays.copyOf(hashedId, 64), IdType.DHT_VALUE);
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException("SHA3-512 algorithm not available.", e);
        }
    }

    public static Map getOIDs(String database, String schema, String table) throws SQLException {
        Map oidMap = new TreeMap<>();
        String query = "SELECT d.oid AS database_oid, n.oid AS schema_oid, c.oid AS table_oid FROM pg_database d "
                + "JOIN pg_namespace n ON n.nspname = ? "
                + "JOIN pg_class c ON c.relnamespace = n.oid AND c.relname = ? "
                + "WHERE d.datname = ?";
        int replicaId = Integer.valueOf(System.getenv("NODE_ID"));
        Connection[] connArr = DatabaseConnector.getConnection(parseCluster(clusterName));
        try (Connection conn = connArr[replicaId];
             PreparedStatement statement = conn.prepareStatement(query)) {
            statement.setString(1, schema);
            statement.setString(2, table);
            statement.setString(3, database);
            try (ResultSet resultSet = statement.executeQuery()) {
                if (resultSet.next()) {
                    oidMap.put("databaseOid", resultSet.getLong("database_oid"));
                    oidMap.put("schemaOid", resultSet.getLong("schema_oid"));
                    oidMap.put("tableOid", resultSet.getLong("table_oid"));
                }
            }
        }
        return oidMap;
    }

    public static String createFullClassAKey(String networkId, MemberOrClassId memberOrClassId, String database, String schema, String table, String record) throws SQLException {
        Map oidMap = getOIDs(database, schema, table);
        return generateId(IdType.RECORD_CLASS_A, networkId, memberOrClassId, Long.toHexString(oidMap.get("databaseOid")),
                Long.toHexString(oidMap.get("schemaOid")), Long.toHexString(oidMap.get("tableOid")), record).toHexString();
    }

    public static String createFullClassUKey(String networkId, MemberOrClassId memberOrClassId, String database, String schema, String table) throws SQLException {
        Map oidMap = getOIDs(database, schema, table);
        return generateId(IdType.FULL_DHT_TX_BLDER_META, networkId, memberOrClassId, Long.toHexString(oidMap.get("databaseOid")),
                Long.toHexString(oidMap.get("schemaOid")), Long.toHexString(oidMap.get("tableOid")), null).toHexString();
    }

    public static String createFullBFTClassURKey(String networkId, MemberOrClassId memberOrClassId, String database, String schema, String table, String record) throws SQLException {
        Map oidMap = getOIDs(database, schema, table);
        return generateId(IdType.RECORD_BFT_CLASS_UR, networkId, memberOrClassId, Long.toHexString(oidMap.get("databaseOid")),
                Long.toHexString(oidMap.get("schemaOid")), Long.toHexString(oidMap.get("tableOid")), record).toHexString();
    }
    
    public static String createFullDHTClassURKey(String networkId, MemberOrClassId memberOrClassId, String database, String schema, String table, String record) throws SQLException {
        Map oidMap = getOIDs(database, schema, table);
        return generateId(IdType.RECORD_DHT_CLASS_UR, networkId, memberOrClassId, Long.toHexString(oidMap.get("databaseOid")),
                Long.toHexString(oidMap.get("schemaOid")), Long.toHexString(oidMap.get("tableOid")), record).toHexString();
    }
    
 // Method to generate a new ID and return the result
    public static String createNewId(IdType idType, String networkId, MemberOrClassId memberOrClassId, String databaseOid, String schemaOid, String tableOid, String recordOid) throws NoSuchAlgorithmException, InterruptedException, ExecutionException, TimeoutException {
        System.out.println("We got this far - beginning to create new Id");

        String identifier = memberOrClassId.getIdentifier();
        IdentifierType identifierType = memberOrClassId.getType();

        System.out.printf("IdentifierType = %s%n", identifierType);

        switch (identifierType) {
            case BASE58:
                if (!isValidBase58(identifier)) {
                    throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 44-character Base58 string.");
                }
                break;
            case HEX:
                if (identifier.length() != 7 || !identifier.matches("[0-9A-Fa-f]+")) {
                    throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 7-character hexadecimal value.");
                }
                break;
            default:
                throw new IllegalArgumentException("Unsupported IdentifierType.");
        }

        // Generate the prefix for the ID
        byte[] prefix = generateIdPrefix(idType, networkId, memberOrClassId, databaseOid, schemaOid, tableOid);

        // Generate the concatenated hashes based on the prefix
        IdTypeBitLengths bitLengths = getBitLengthsForIdType(idType);
        if (bitLengths == null) {
            throw new IllegalArgumentException("Unknown or unsupported idType.");
        }
        TreeMap hashResult = generateConcatenatedHashesWithPrefix(prefix, bitLengths, databaseOid, schemaOid, tableOid, recordOid);

        // Combine the hash sections into a single ID
        StringBuilder fullId = new StringBuilder();
        hashResult.forEach((index, value) -> fullId.append("-").append(value));
        
     // Store the ID in etcd
        
        storeIdInEtcd(idType, fullId.toString());

        System.out.println("Successfully created new Id!");
        return fullId.toString();
    }
    
    public static String createNewIdNonDb(IdType idType, bftsmart.demo.itotcca.Cluster networkId2, MemberOrClassId memberOrClassId) 
            throws NoSuchAlgorithmException, InterruptedException, ExecutionException, TimeoutException {
        
        System.out.println("We got this far - beginning to create new Id");

        if (networkId2 == null || memberOrClassId == null) {
            throw new IllegalArgumentException("Network ID and Member/Class ID cannot be null.");
        }

        String identifier = memberOrClassId.getIdentifier();
        IdentifierType identifierType = memberOrClassId.getType();

        System.out.printf("IdentifierType = %s%n", identifierType);

        System.out.printf("💡 createNewIdNonDb: identifier = %s, length = %d%n",
        	    memberOrClassId.getIdentifier(),
        	    memberOrClassId.getIdentifier().length()
        	);
        
        switch (identifierType) {
            case BASE58:
                if (!isValidBase58(identifier)) {
                    throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 44-character Base58 string.");
                }
                break;
            case HEX:
                if (identifier.length() != 7 || !identifier.matches("[0-9A-Fa-f]+")) {
                    throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 7-character hexadecimal value.");
                }
                break;
            default:
                throw new IllegalArgumentException("Unsupported IdentifierType.");
        }

        // ✅ Generate a **random** Id using the correct method
        Id randomId = Id.random(idType, networkId2, memberOrClassId);
        if (randomId == null) {
            throw new RuntimeException("Failed to generate an ID. randomId is null.");
        }

        // ✅ Convert the Id to a **hex string**
        String randomIdHex = randomId.toHexString();
        if (randomIdHex == null || randomIdHex.isEmpty()) {
            throw new RuntimeException("Generated ID is null or empty.");
        }
        
        System.out.println("Successfully created new random Id: " + randomIdHex);

        // ✅ Store the ID in etcd
        String key = storeIdInEtcd(idType, randomIdHex);
        
        if (key == null) {
            throw new IllegalStateException("Failed to store ID in etcd: key is null.");
        }

        System.out.println("Successfully stored ID in etcd, with key: " + key);
        
        return randomIdHex;
    }
    
 // Store ID in etcd
    public static String storeIdInEtcd(IdType idType, String id) throws InterruptedException, ExecutionException, TimeoutException {
        KV kvClient = etcdClient.getKVClient();

        // 🔹 Generate the etcd key using a switch expression
        String key = switch (idType) {
            case PEER_NODE_SUPERNODE -> "/config/ids/peer/supernode/" + getContainerName();
            case PEER_NODE_SECONDARY_NODE -> "/config/ids/peer/secondary/" + getContainerName();
            //case PEER_NODE_TERTIARY_NODE -> "/config/ids/peer/tertiary/" + getDeviceUniqueId();
            case DATABASE -> "/config/ids/database/" + getDatabaseKey(id);
            case SCHEMA -> "/config/ids/schema/" + getSchemaKey(id);
            case TABLE_CLASS_A, TABLE_BFT_CLASS_UR -> "/config/ids/table/" + getTableKey(id);
            case RECORD_CLASS_A, RECORD_BFT_CLASS_UR, RECORD_DHT_CLASS_UR -> "/config/ids/record/" + getRecordKey(id);
            case FULL_CLASS_A_KEY -> "/config/ids/full/classA/" + getFullClassAKey(id);
            case FULL_BFT_CLASS_UR_KEY -> "/config/ids/full/bftclassUR/" + getFullBFTClassURKey(id);
            case FULL_DHT_CLASS_UR_KEY -> "/config/ids/full/dhtclassUR/" + getFullDHTClassURKey(id);
            case FULL_BFT_CLASS_U_KEY -> "/config/ids/full/classU/" + getFullBFTClassUKey(id);
            case FULL_DHT_TX_BLDER_META -> "/config/ids/dht/blder/" + getDhtTxBlderMetaKey(id);
            case DHT_VALUE, ROLE, CLIENT_PERSONAL -> "/config/ids/" + idType.name().toLowerCase() + "/" + id;
            default -> "/config/ids/unknown";
        };

        // Convert key and value to ByteSequence
        ByteSequence keyByteSequence = ByteSequence.from(key, StandardCharsets.UTF_8);
        ByteSequence valueByteSequence = ByteSequence.from(id, StandardCharsets.UTF_8);

        boolean registered = false;
        int retries = 10;

        while (!registered && retries > 0) {
            try {
                kvClient.put(keyByteSequence, valueByteSequence).get(); // Store value in etcd

                // Verify storage
                CompletableFuture getFuture = kvClient.get(keyByteSequence);
                GetResponse getResponse = getFuture.get();

                registered = !getResponse.getKvs().isEmpty();

                if (registered) {
                    System.out.printf("✅ ID successfully stored in etcd with key = %s%n", key);
                    return key;
                }

            } catch (Exception e) {
                System.err.printf("❌ Failed to register ID (attempt %d/10), retrying...%n", (11 - retries));
                e.printStackTrace();
            }

            retries--;
            Thread.sleep(5000);  // Ensure sleep is not interrupted
        }

        throw new IllegalStateException("❌ Failed to register ID after 10 attempts.");
    }

    public static String storeValueInEtcd(Integer entnum, Long slabId, byte[] valueByteArray) throws InterruptedException {
        if (etcdClient == null) {
            etcdClient = initializeEtcdClient();
        }

        KV kvClient = etcdClient.getKVClient();

		UUID txId = UUID.randomUUID();
		// 🔹 Generate the etcd key dynamically
        String key = String.format("/metadata/values/txId/%s/networkId/%s/entnum/%d/databaseId/%s/schemaId/%s/tableId/%s/slabId/%d/blockId/%s",
        	    txId, networkId, entnum, databaseId, schemaId, tableId, slabId, blockId);

        // Convert key and value to ByteSequence
        ByteSequence keyByteSequence = ByteSequence.from(key, StandardCharsets.UTF_8);
        ByteSequence valueByteSequence = ByteSequence.from(valueByteArray);  // ✅ Corrected

        boolean registered = false;
        int retries = 10;

        while (!registered && retries > 0) {
            try {
                // 🔹 Store value in etcd
                kvClient.put(keyByteSequence, valueByteSequence).get();

                // 🔹 Verify storage
                CompletableFuture getFuture = kvClient.get(keyByteSequence);
                GetResponse getResponse = getFuture.get();

                registered = !getResponse.getKvs().isEmpty();

                if (registered) {
                    System.out.printf("✅ Value successfully stored in etcd with key = %s%n", key);
                    return key;
                }

            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();  // ✅ Preserve thread interruption state
                throw new IllegalStateException("❌ Interrupted while registering value in etcd.", e);
            } catch (Exception e) {
                System.err.printf("❌ Failed to register value (attempt %d/10), retrying...%n", (11 - retries));
                e.printStackTrace();
            }

            retries--;
            Thread.sleep(5000);  // ✅ Ensure sleep is not interrupted
        }

        throw new IllegalStateException("❌ Failed to register value in etcd after 10 attempts.");
    }


    
    public static String getEtcdKey(IdType idType, String id) throws InterruptedException, ExecutionException, TimeoutException {
    	
    	// 🔹 Generate the etcd key using a switch expression
        String key = switch (idType) {
            case PEER_NODE_SUPERNODE -> "/config/ids/peer/supernode/" + getContainerName();
            case PEER_NODE_SECONDARY_NODE -> "/config/ids/peer/secondary/" + getContainerName();
            //case PEER_NODE_TERTIARY_NODE -> "/config/ids/peer/tertiary/" + getDeviceUniqueId();
            case DATABASE -> "/config/ids/database/" + getDatabaseKey(id);
            case SCHEMA -> "/config/ids/schema/" + getSchemaKey(id);
            case TABLE_CLASS_A -> "/config/ids/table/classA/" + getTableKey(id);
            case TABLE_BFT_CLASS_UR -> "/config/ids/table/classUR/" + getTableKey(id);
            case DHT_TX_BLDER_TABLE -> "/config/ids/table/dhtclassU/" + getTableKey(id);
            case TABLE_BFT_CLASS_U -> "/config/ids/table/bftclassU/" + getTableKey(id);
            case RECORD_CLASS_A -> "/config/ids/record/classA/" + getRecordKey(id);
            case RECORD_BFT_CLASS_UR -> "/config/ids/record/classUR/" + getRecordKey(id);
            case FULL_CLASS_A_KEY -> "/config/ids/full/classA/" + getFullClassAKey(id);
            case FULL_BFT_CLASS_UR_KEY -> "/config/ids/full/bftclassUR/" + getFullBFTClassURKey(id);
            case FULL_DHT_CLASS_UR_KEY -> "/config/ids/full/dhtclassUR/" + getFullDHTClassURKey(id);
            case FULL_BFT_CLASS_U_KEY -> "/config/ids/full/classU/" + getFullBFTClassUKey(id);
            case FULL_DHT_TX_BLDER_META -> "/config/ids/dht/blder/" + getDhtTxBlderMetaKey(id);
            case ROLE, CLIENT_PERSONAL -> "/config/ids/" + idType.name().toLowerCase() + "/" + id;
            case DHT_VALUE -> String.format("/metadata/values/txId/%s/networkId/%s/entnum/%d/databaseId/%s/schemaId/%s/tableId/%s/slabId/%d/blockId/%s",
            	    txId, networkId, entnum, databaseId, schemaId, tableId, slabId, blockId);
            case NETWORK_ID -> "/config/ids/network/" + getNetworkName();
            case MEMBER_OR_CLASS_ID -> "/config/ids/class/" + getDatabaseName();
            default -> "/config/ids/unknown";
        };
		return key;

    }
    
    private static String getDatabaseName() {
		String dbName = System.getenv("DATABASE_NAME");
		return dbName;
	}


    public static String getNetworkName() {
        String clusterEnv = System.getenv("CLUSTER");
        if (clusterEnv == null) {
            throw new IllegalStateException("CLUSTER environment variable is not set.");
        }

        String baseName = clusterEnv.startsWith("SUPER")
                ? clusterEnv.substring("SUPER".length())
                : clusterEnv;

        return baseName.toLowerCase();
    }

	public static String getContainerName() {
        
        if (getClusterNameLowerCase().contains("super")) {
        	String clusterNameWithoutSuper = getClusterNameLowerCase().replaceFirst("super", "");
        	containerName = "bft-dht-" + clusterNameWithoutSuper + "-super";
        } else {
        	containerName = "bft-dht-" + getClusterNameLowerCase() + "-" + System.getenv("NODE_ID");
        }
		return containerName;
    }
	
	public static bftsmart.demo.itotcca.Cluster extractClusterFromId(Id id) {
	    String idHex = id.toHex();
	    String clusterPrefix = idHex.substring(0, 7); // Adjust bits/length as appropriate
	    return bftsmart.demo.itotcca.Cluster.fromId7(clusterPrefix);
	}

        
    // Dummy mappings for database, schema, table, and record names
    private static final Map databaseMap = new TreeMap<>();
    private static final Map schemaMap = new TreeMap<>();
    private static final Map tableMap = new TreeMap<>();
    
    static {
        // Populate dummy mappings
        databaseMap.put("1a2b3c", "EnterpriseDB");
        schemaMap.put("4d5e6f", "FinanceSchema");
        tableMap.put("7g8h9i", "TransactionsTable");
    }
    
    
    
    public static String[] resolveKeyToTableAndRecord(String fullDHTKey) {
        // Remove the 8-bit prefix
        String baseKey = fullDHTKey.substring(8);

        // Extracting different parts of the key based on bit allocations
        String networkId = baseKey.substring(0, 28);
        String memberOrClassId = baseKey.substring(28, 56);
        String databaseId = baseKey.substring(56, 154);
        String schemaId = baseKey.substring(154, 254);
        String tableId = baseKey.substring(254, 354);
        String recordId = baseKey.substring(354, 504);  // Keep for direct use

        // Fetch names from etcd
        String databaseName = getDatabaseFromId("/config/mappings/database/" + databaseId);
        String schemaName = getSchemaFromId("/config/mappings/schema/" + schemaId);
        String tableName = getTableFromId("/config/mappings/table/" + tableId);

        if (databaseName == null || schemaName == null || tableName == null) {
            System.err.println("❌ Missing mapping for key: " + fullDHTKey);
            return null;
        }

        // Return human-readable values for processing
        return new String[]{databaseName, schemaName, tableName, recordId};
    }
    
    
    public static String resolveTableAndRecordToKey(String fullTableNamePlusRecordKey) {
        // Remove the 8-bit prefix
        String mappedNameAt = fullTableNamePlusRecordKey;

        // Extracting different parts of the key based on bit allocations
        String networkId = mappedNameAt.substring(0, 28);
        String memberOrClassId = mappedNameAt.substring(28, 56);
        String databaseName = mappedNameAt.substring(56, 154);
        String schemaName = mappedNameAt.substring(154, 254);
        String tableName = mappedNameAt.substring(254, 354);
        String recordId = mappedNameAt.substring(354, 504);  // Keep for direct use

        // Fetch names from etcd
        String databaseId = getIdFromDatabaseName("/config/mappings/databaseNames/" + databaseName);
        String schemaId = getIdFromSchemaName("/config/mappings/schemaNames/" + schemaName);
        String tableId = getIdFromTableName("/config/mappings/tableNames/" + tableName);

        if (databaseId == null || schemaId == null || tableId == null) {
            System.err.println("❌ Missing mapping for key: " + fullTableNamePlusRecordKey);
            return null;
        }

        // Return human-readable values for processing
        return "0F" + networkId + memberOrClassId + databaseId + schemaId + tableId + recordId;
    }
    
    
    
    public static void storeIdNameMappings(String databaseId, String databaseName, 
            String schemaId, String schemaName, 
            String tableId, String tableName) {
			// Store ID -> Name mappings
			putToEtcd("/config/mappings/database/" + databaseId, databaseName.getBytes(StandardCharsets.UTF_8));
			putToEtcd("/config/mappings/schema/" + schemaId, schemaName.getBytes(StandardCharsets.UTF_8));
			putToEtcd("/config/mappings/table/" + tableId, tableName.getBytes(StandardCharsets.UTF_8));
			
			// Store Name -> ID mappings (for reverse lookup)
			putToEtcd("/config/mappings/databaseNames/" + databaseName, databaseId.getBytes(StandardCharsets.UTF_8));
			putToEtcd("/config/mappings/schemaNames/" + schemaName, schemaId.getBytes(StandardCharsets.UTF_8));
			putToEtcd("/config/mappings/tableNames/" + tableName, tableId.getBytes(StandardCharsets.UTF_8));
	}

    
    

    
    public static String getDatabaseFromId(String databaseId) {
        String lookupKey = "/config/mappings/database/" + databaseId;
        return fetchEtcdMapping(lookupKey);
    }

    private static String getSchemaFromId(String schemaId) {
        String lookupKey = "/config/mappings/schema/" + schemaId;
        return fetchEtcdMapping(lookupKey);
    }

    private static String getTableFromId(String tableId) {
        String lookupKey = "/config/mappings/table/" + tableId;
        return fetchEtcdMapping(lookupKey);
    }

    // 🔹 Fetch mapping from etcd
    private static String fetchEtcdMapping(String key) {
        byte[] resultBytes = getFromEtcd(key);
        return (resultBytes != null) ? new String(resultBytes, StandardCharsets.UTF_8) : "UNKNOWN";
    }
    
    
    
    public static String getIdFromDatabaseName(String databaseName) {
        String lookupKey = "/config/mappings/databaseNames/" + databaseName;
        return fetchEtcdMapping(lookupKey);
    }

    private static String getIdFromSchemaName(String schemaName) {
        String lookupKey = "/config/mappings/schemaNames/" + schemaName;
        return fetchEtcdMapping(lookupKey);
    }

    private static String getIdFromTableName(String tableName) {
        String lookupKey = "/config/mappings/tableNames/" + tableName;
        return fetchEtcdMapping(lookupKey);
    }
    
    

    
    
    private static final ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 50061)
            .usePlaintext()
            .build();

    private static final KeyLookupServiceGrpc.KeyLookupServiceBlockingStub keyLookupStub =
            KeyLookupServiceGrpc.newBlockingStub(channel);

    // 🔹 Replace the direct method calls with gRPC requests to key-lookup-service
    public static String getDatabaseKey(String fullId) {
        KeyLookupRequest request = KeyLookupRequest.newBuilder()
                .setFullId(fullId)
                .build();
        KeyLookupResponse response = keyLookupStub.lookupDatabaseKey(request);
        return response.getDatabaseKey();
    }

    public static String getSchemaKey(String fullId) {
        KeyLookupRequest request = KeyLookupRequest.newBuilder()
                .setFullId(fullId)
                .build();
        KeyLookupResponse response = keyLookupStub.lookupSchemaKey(request);
        return response.getSchemaKey();
    }

    public static String getTableKey(String fullId) {
        KeyLookupRequest request = KeyLookupRequest.newBuilder()
                .setFullId(fullId)
                .build();
        KeyLookupResponse response = keyLookupStub.lookupTableKey(request);
        return response.getTableKey();
    }

    public static String getRecordKey(String fullId) {
        KeyLookupRequest request = KeyLookupRequest.newBuilder()
                .setFullId(fullId)
                .build();
        KeyLookupResponse response = keyLookupStub.lookupRecordKey(request);
        return response.getRecordKey();
    }
    
    public static String getFullClassAKey(String fullId) {
        return replaceKeyType(fullId, "00010001"); // 110 for Full-Class Keys
    }

    public static String getFullBFTClassURKey(String fullId) {
        return replaceKeyType(fullId, "00010100");
    }
    
    
    public static String getFullDHTClassURKey(String fullId) {
        return replaceKeyType(fullId, "00010101");
    }
    
    public static String getFullBFTClassUKey(String fullId) {
        return replaceKeyType(fullId, "00010010");
    }

    public static String getDhtTxBlderMetaKey(String fullId) {
        return replaceKeyType(fullId, "00010011");
    }

    public static String getDhtTxTraceMetaKey(String fullId) {
        return replaceKeyType(fullId, "00010101");
    }


    private static String replaceKeyType(String fullId, String newKeyType) {
        return newKeyType + fullId.substring(8); // Replace first 8 bits only
    }

	// Generate a unique client ID based on UUID or existing DHT ID
    private static String getClientUniqueId() {
        return UUID.randomUUID().toString(); // Alternatively, fetch from DHT
    }
    
 /** Generate a unique device ID based on UUID or existing DHT ID
    public static String getDeviceUniqueId() throws InterruptedException, ExecutionException, TimeoutException {
    	if (getDeviceUniqueId() == null) {
    		return UUID.randomUUID().toString(); // Alternatively, fetch from DHT
    } else {
    	String deviceUniqueId = IdManager.getDeviceUniqueId();
        String keyPath = "/config/configs/peer/tertiary/" + IdManager.getDeviceUniqueId();

        Client etcdClient = initializeEtcdClient();
        ByteSequence key = ByteSequence.from(keyPath, StandardCharsets.UTF_8);
        CompletableFuture future = etcdClient.getKVClient().get(key);
        GetResponse response = future.get(3, TimeUnit.SECONDS);

        if (response.getKvs().isEmpty()) {
            throw new IllegalStateException("❌ No config found for key: " + keyPath);
        }

        String configData = response.getKvs().get(0).getValue().toString(StandardCharsets.UTF_8);
        // Parse configData to create DefaultConfiguration if you serialize JSON

        throw new UnsupportedOperationException("Implement JSON deserialization as needed.");
    }
    }**/

	public static TreeMap generateHashes(IdType idType, String networkId, MemberOrClassId memberOrClassId) throws NoSuchAlgorithmException {
        // Validate inputs
        if (networkId.length() != 7 || !networkId.matches("[0-9A-Fa-f]+")) {
            throw new IllegalArgumentException("Invalid networkId. Must be a 7-character hexadecimal value.");
        }

        boolean isBase58 = networkId.equals("0000000") && idType == IdType.CLIENT_PERSONAL;
        if (isBase58) {
            if (!isValidBase58(memberOrClassId.toString())) {
                throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 44-character Base58 string.");
            }
        } else {
            if (memberOrClassId.toString().length() != 7 || !memberOrClassId.toString().matches("[0-9A-Fa-f]+")) {
                throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 7-character hexadecimal value.");
            }
        }

        // Generate the prefix
        byte[] prefix = generateIdPrefix(idType, networkId, memberOrClassId, databaseOid, schemaOid, tableOid);

        // Get bit lengths for the ID type
        IdTypeBitLengths bitLengths = getBitLengthsForIdType(idType);
        if (bitLengths == null) {
            throw new IllegalArgumentException("Unknown or unsupported idType.");
        }

        // Generate and return hashes
        return generateConcatenatedHashesWithPrefix(prefix, bitLengths, databaseOid, schemaOid, tableOid, recordOid);
    }
	
	public static class IdTypeBitLengths {
	    int[] bitLengths;
	    boolean forcePublicSchema;

	    public IdTypeBitLengths(int[] bitLengths, boolean forcePublicSchema) {
	        this.bitLengths = bitLengths;
	        this.forcePublicSchema = forcePublicSchema;
	    }
	}

	private static IdTypeBitLengths getBitLengthsForIdType(IdType idType) {
	    boolean forcePublicSchema = (idType == IdType.FULL_DHT_TX_BLDER_META); // ✅ Dynamic schema decision

	    return switch (idType) {
	        case PEER_NODE_SUPERNODE, PEER_NODE_SECONDARY_NODE, PEER_NODE_TERTIARY_NODE, ROLE, DHT_VALUE ->
	            new IdTypeBitLengths(new int[]{64, 448}, false);
	        case DATABASE -> 
	            new IdTypeBitLengths(new int[]{64, 98}, false);
	        case SCHEMA -> 
	            new IdTypeBitLengths(new int[]{64, 100}, forcePublicSchema); // ✅ Now dynamic!
	        case TABLE_CLASS_A, TABLE_BFT_CLASS_UR, TABLE_BFT_CLASS_U, TABLE_DHT_CLASS_UR -> 
	            new IdTypeBitLengths(new int[]{64, 100}, false);
	        case FULL_BFT_CLASS_U_KEY -> 
	            new IdTypeBitLengths(new int[]{64, 98, 100, 100}, false);
	        case RECORD_CLASS_A, RECORD_BFT_CLASS_UR, RECORD_DHT_CLASS_UR ->
	            new IdTypeBitLengths(new int[]{64, 150}, false);
	        case FULL_CLASS_A_KEY, FULL_BFT_CLASS_UR_KEY, FULL_DHT_CLASS_UR_KEY -> 
	            new IdTypeBitLengths(new int[]{64, 98, 100, 100, 150}, false);
	        case FULL_DHT_TX_BLDER_META -> 
	            new IdTypeBitLengths(new int[]{64, 98, 100, 100}, true); // ✅ Always public
	        case DHT_TX_BLDER_RECORD ->
	            new IdTypeBitLengths(new int[]{64, 100}, false); // ✅ No schema concern → false
	        case CLIENT_PERSONAL -> 
	            new IdTypeBitLengths(new int[]{64, 448}, false);
	        case NETWORK_ID ->
	        	new IdTypeBitLengths(new int[]{8, 28}, false);
	        case MEMBER_OR_CLASS_ID ->
	        	new IdTypeBitLengths(new int[]{8, 28}, false);
	        default -> null;
	    };
	}
    
	private static byte[] generateNonDbIdPrefix(IdType idType, bftsmart.demo.itotcca.Cluster networkId2, MemberOrClassId memberOrClassId) {
	    try {
	        MessageDigest sha3 = MessageDigest.getInstance("SHA3-512");

	        // Generate a unique identifier based on network ID and node ID
	        String prefixInput = idType.getHexPrefix() + "-" + networkId2 + "-" + memberOrClassId.getIdentifier();
	        byte[] hash = sha3.digest(prefixInput.getBytes(StandardCharsets.UTF_8));

	        return Arrays.copyOf(hash, 8); // Take first 8 bytes for the prefix (64 bits)
	    } catch (NoSuchAlgorithmException e) {
	        throw new RuntimeException("SHA3-512 algorithm not available.", e);
	    }
	}
	
	private static byte[] generateIdPrefix(IdType idType, String networkId, MemberOrClassId memberOrClassId, 
            String databaseOid, String schemaOid, String tableOid) {
		try {
		MessageDigest sha3 = MessageDigest.getInstance("SHA3-512");
		
		// Ensure all OIDs are properly formatted (pad with leading zeros)
		databaseOid = String.format("%7s", databaseOid).replace(' ', '0');
		schemaOid = String.format("%7s", schemaOid).replace(' ', '0');
		tableOid = String.format("%7s", tableOid).replace(' ', '0');
		
		// Concatenate components for hashing
		String input = idType.getHexPrefix() + "-" + networkId + "-" + memberOrClassId.getIdentifier() + 
		"-" + databaseOid + "-" + schemaOid + "-" + tableOid;
		
		// Compute SHA3-512 hash
		byte[] hash = sha3.digest(input.getBytes(StandardCharsets.UTF_8));
		
		// Extract the first 8 bytes (64-bit prefix)
		return Arrays.copyOf(hash, 8);
		} catch (NoSuchAlgorithmException e) {
		throw new RuntimeException("SHA3-512 algorithm not available.", e);
		}
	}

	private static TreeMap generateConcatenatedHashesWithPrefix(
	        byte[] prefix, IdTypeBitLengths bitLengthsInfo, 
	        String databaseOid, String schemaOid, String tableOid, String recordOid) throws NoSuchAlgorithmException {
	    
	    // Convert prefix from byte[] to Hex
	    String prefixHex = bytesToHex(prefix);

	    // Ensure OIDs are not null to prevent NullPointerException
	    databaseOid = (databaseOid != null) ? databaseOid : "";
	    schemaOid = (schemaOid != null) ? schemaOid : "";
	    tableOid = (tableOid != null) ? tableOid : "";
	    recordOid = (recordOid != null) ? recordOid : "";

	    // Concatenate prefix + OIDs for hashing
	    String fullInput = prefixHex + "-" + databaseOid + "-" + schemaOid + "-" + tableOid + "-" + recordOid;

	    // Create SHA3-512 hash
	    MessageDigest digest = MessageDigest.getInstance("SHA3-512");
	    byte[] hash = digest.digest(fullInput.getBytes(StandardCharsets.UTF_8));

	    // Extract bit lengths array
	    int[] bitLengths = bitLengthsInfo.bitLengths;

	    // Create map for hash sections
	    TreeMap hashSections = new TreeMap<>();
	    int byteIndex = 0;

	    for (int bitLength : bitLengths) {
	        int byteLength = bitLength / 8;

	        // Ensure we do not exceed the hash length
	        if (byteIndex + byteLength > hash.length) {
	            break;  // Avoid `ArrayIndexOutOfBoundsException`
	        }

	        // Extract and store hash segment
	        String segment = bytesToHex(Arrays.copyOfRange(hash, byteIndex, byteIndex + byteLength));
	        hashSections.put(byteIndex, segment);

	        byteIndex += byteLength;
	    }

	    return hashSections;
	}

	  
    
    private static String bytesToHex(byte[] bytes) {
        BigInteger bigInteger = new BigInteger(1, bytes);
        StringBuilder hexString = new StringBuilder(bigInteger.toString(16));
        while (hexString.length() < bytes.length * 2) {
            hexString.insert(0, '0');
        }
        return hexString.toString();
    }

    private static boolean isValidBase58(String value) {
        final String BASE58_ALPHABET = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";
        for (char c : value.toCharArray()) {
            if (BASE58_ALPHABET.indexOf(c) == -1) {
                return false;
            }
        }
        return value.length() == 44;
    }

    // Helper: Convert bytes to binary string
    public static String bytesToBinaryString(byte[] bytes) {
        StringBuilder binary = new StringBuilder();
        for (byte b : bytes) {
            binary.append(String.format("%8s", Integer.toBinaryString(b & 0xFF)).replace(' ', '0'));
        }
        return binary.toString();
    }
    
    public static Id createHashedId(String input, IdType type) {
        return new Id(Id.hashValue(input), type);
    }

    public static byte[] uuidToByteArray(UUID uuid) {
        ByteBuffer buffer = ByteBuffer.allocate(16);
        buffer.putLong(uuid.getMostSignificantBits());
        buffer.putLong(uuid.getLeastSignificantBits());
        return buffer.array();
    }
    
    public static boolean isValidId(Id id, IdType expectedType) {
        return id != null && id.getType() == expectedType;
    }

    public static Id computeDistance(Id id1, Id id2) {
        byte[] resultBytes = new byte[Id.SIZE];
        byte[] bytes1 = id1.getBytes();
        byte[] bytes2 = id2.getBytes();

        for (int i = 0; i < Id.SIZE; i++) {
            resultBytes[i] = (byte) (bytes1[i] ^ bytes2[i]);
        }
        return new Id(resultBytes, id1.getType());
    }
    
    public static Id of(Prefix prefix) {
	    if (prefix == null) {
	        throw new IllegalArgumentException("Prefix cannot be null.");
	    }

	    // Create a new Id using the bytes from the Prefix and its depth
	    byte[] prefixBytes = prefix.getBytes(); // Assuming Prefix has a getBytes() method
	    int depth = prefix.getDepth(); // Assuming Prefix has a getDepth() method

	    // Copy the bytes from the prefix and adjust the remaining bits based on depth
	    byte[] idBytes = new byte[Id.SIZE];
	    System.arraycopy(prefixBytes, 0, idBytes, 0, prefixBytes.length);

	    // Set remaining bits to 0 beyond the depth
	    int byteIndex = depth / 8;
	    int bitIndex = depth % 8;

	    if (byteIndex < idBytes.length) {
	        idBytes[byteIndex] &= ~(0xFF >> bitIndex); // Zero out bits beyond the depth
	    }

	    for (int i = byteIndex + 1; i < idBytes.length; i++) {
	        idBytes[i] = 0; // Zero out all bytes beyond the depth
	    }

	    return new bftsmart.demo.itotcca.IdManager.Id(idBytes, IdType.DHT_VALUE);
	}
    
    public static IdManager.Id fromId(Id id) {
        // Conversion logic from Id to IdManager.Id
        return new IdManager.Id(id.getBytes(), IdType.DHT_VALUE); // Example
    }

    public static byte[] hashValue(String nodeSeed) {
        try {
            // Use SHA3-512 for strong hashing
            MessageDigest digest = MessageDigest.getInstance("SHA3-512");
            return digest.digest(nodeSeed.getBytes());
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException("SHA3-512 algorithm not available.", e);
        }
    }
    
    public static bftsmart.demo.itotcca.Cluster getNetworkIdFromEnv() {
    	
    	System.out.println("[DEBUG] Raw networkName = " + getNetworkName());
    	
        String networkName = getNetworkName();

        if (networkName == null || networkName.isEmpty()) {
            throw new IllegalStateException("CLUSTER environment variable is not set.");
        }

        try {
            // Hash using SHA-256
            MessageDigest digest = MessageDigest.getInstance("SHA-256");
            byte[] hash = digest.digest(networkName.getBytes(StandardCharsets.UTF_8));

            // Convert first 4 bytes to hex string (8 hex chars), then truncate to 7
            StringBuilder hexString = new StringBuilder();
            for (int i = 0; i < 4; i++) {
                String hex = Integer.toHexString(0xff & hash[i]);
                if (hex.length() == 1) hexString.append('0');
                hexString.append(hex);
            }

            String id7 = hexString.toString().substring(0, 7).toUpperCase();

            // Match the 7-character hex ID to an enum's identifier field
            for (bftsmart.demo.itotcca.Cluster id : bftsmart.demo.itotcca.Cluster.values()) {
            	if (id.getId7().equalsIgnoreCase(id7)) {
                    return id;
                }
            }

            throw new IllegalStateException("No matching Cluster found for: " + id7);

        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException("SHA-256 not available", e);
        }
    }
    
    public static MemberOrClassId getMemberOrClassIdFromEnv() {
        String dbName = System.getenv("DATABASE_NAME");

        if (dbName == null || dbName.isEmpty()) {
            throw new IllegalStateException("DATABASE_NAME environment variable is not set.");
        }

        try {
            return MemberOrClassId.parse(dbName); // 👈 Use the custom method
        } catch (IllegalArgumentException e) {
            throw new IllegalStateException("No matching MemberOrClassId for: " + dbName);
        }
    }

    
    public static bftsmart.demo.itotcca.IdManager.Id createIdFromString(String key) {
        if (key == null || key.isEmpty()) {
            throw new IllegalArgumentException("Key cannot be null or empty.");
        }

        try {
            // Generate a hashed ID from the key using SHA-256
            MessageDigest digest = MessageDigest.getInstance("SHA-256");
            byte[] hashBytes = digest.digest(key.getBytes(StandardCharsets.UTF_8));

            // Truncate or pad the hash to match the required ID length
            byte[] idBytes = Arrays.copyOf(hashBytes, Id.ID_BYTE_LENGTH);

            return new bftsmart.demo.itotcca.IdManager.Id(idBytes, IdType.DHT_VALUE);
        } catch (NoSuchAlgorithmException e) {
            throw new IllegalStateException("SHA-256 algorithm not found.", e);
        }
    }
    /**
     * Generates a **512-bit ID (FULL_CLASS_A_KEY)** for Database + Schema + Table + Record.
     */
    public static Id generateFullClassAKey(String networkId, String databaseOid, String schemaOid, String tableOid, String recordOid) throws NoSuchAlgorithmException {
        String prefix = IdType.FULL_CLASS_A_KEY.getHexPrefix();
        return generateFullKey(prefix, networkId, databaseOid, schemaOid, tableOid, recordOid);
    }

    /**
     * Generates a **5122-bit ID (FULL_CLASS_UR_KEY)** for Database + Schema + Table + Record.
     */
    public static Id generateFullBFTClassURKey(String networkId, String databaseOid, String schemaOid, String tableOid, String recordOid) throws NoSuchAlgorithmException {
        String prefix = IdType.FULL_BFT_CLASS_UR_KEY.getHexPrefix();
        return generateFullKey(prefix, networkId, databaseOid, schemaOid, tableOid, recordOid);
    }

    /**
     * Generates a **512-bit ID (FULL_CLASS_UR_KEY)** for Database + Schema + Table + Record.
     */
    public static Id generateFullDHTClassURKey(String networkId, String databaseOid, String schemaOid, String tableOid, String recordOid) throws NoSuchAlgorithmException {
        String prefix = IdType.FULL_DHT_CLASS_UR_KEY.getHexPrefix();
        return generateFullKey(prefix, networkId, databaseOid, schemaOid, tableOid, recordOid);
    }

    
    /**
     * Generates a **362-bit ID (FULL_CLASS_U_KEY)** for Database + Schema + Table.
     */
    public static Id generateFullBFTClassUKey(String networkId, String databaseOid, String schemaOid, String tableOid) throws NoSuchAlgorithmException {
        String prefix = IdType.FULL_BFT_CLASS_U_KEY.getHexPrefix();
        return generateFullKey(prefix, networkId, databaseOid, schemaOid, tableOid, null);
    }
    
    /**
     * Generates a **362-bit ID (FULL_CLASS_U_KEY)** for Database + Schema + Table.
     */
    public static Id generateFullDHTClassUKey(String networkId, String databaseOid, String schemaOid, String tableOid) throws NoSuchAlgorithmException {
        String prefix = IdType.FULL_DHT_TX_BLDER_META.getHexPrefix();
        return generateFullKey(prefix, networkId, databaseOid, schemaOid, tableOid, null);
    }


    /**
     * Generalized Key Generator with Prefix.
     */
    private static Id generateFullKey(String prefix, String networkId, String databaseOid, String schemaOid, String tableOid, String recordOid) throws NoSuchAlgorithmException {
        MessageDigest sha3 = MessageDigest.getInstance("SHA3-512");

        StringBuilder fullKey = new StringBuilder()
                .append(networkId)
                .append(databaseOid)
                .append(schemaOid)
                .append(tableOid);

        if (recordOid != null) {
            fullKey.append(recordOid);
        }

        byte[] hashedId = sha3.digest(fullKey.toString().getBytes(StandardCharsets.UTF_8));
        return new Id(Arrays.copyOf(hashedId, 64), IdType.DHT_VALUE);
    }

    /**
     * Enum for ID types with reorganized prefixes.
     */
    public enum IdType {
        PEER_NODE_SUPERNODE("00"),
        PEER_NODE_SECONDARY_NODE("01"),
        PEER_NODE_TERTIARY_NODE("02"),
        DATABASE("03"),
        SCHEMA("04"),
        TABLE_CLASS_A("05"),
        RECORD_CLASS_A("06"),
        TABLE_BFT_CLASS_U("07"),
        TABLE_BFT_CLASS_UR("08"),
        RECORD_BFT_CLASS_UR("09"),
        TABLE_DHT_CLASS_UR("0A"),
        RECORD_DHT_CLASS_UR("0B"),
        DHT_TX_BLDER_TABLE("0C"),
        DHT_TX_BLDER_RECORD("0D"),
        ROLE("0E"),
        CLIENT_PERSONAL("0F"),
        DHT_VALUE("10"),
        FULL_CLASS_A_KEY("11"),          // 512-bit Key (DB + Schema + Table + Record)
        FULL_BFT_CLASS_U_KEY("12"),      // 362-bit Key (DB + Schema + Table)
        FULL_DHT_TX_BLDER_META("13"),
        FULL_BFT_CLASS_UR_KEY("14"),     // 512-bit Key (DB + Schema + Table + Record)
        FULL_DHT_CLASS_UR_KEY("15"),     // 512-bit Key (DB + Schema + Table + Record)
        NETWORK_ID("16"),
        MEMBER_OR_CLASS_ID("17");

        private final String hexPrefix;

        IdType(String hexPrefix) {
            this.hexPrefix = hexPrefix;
        }

		String getHexPrefix() {
            return this.hexPrefix;
        }

        // Optional: Get prefix as a single byte if needed
        public byte getPrefixByte() {
            return (byte) Integer.parseInt(hexPrefix, 16);
        }

        @Override
        public String toString() {
            return this.name() + "(" + hexPrefix + ")";
        }

 // Method to generate a new Full Class_A ID and return the result
    public static String createNewFullAId(IdType FULL_CLASS_A_KEY, String networkId, MemberOrClassId memberOrClassId, GeneralMessage request) throws Exception {
        
        // Extract database, schema, and table name from request
        String fullTableName = request.getFullTableName(); // Expected format: "database.schema.table"
        
        // Split the full table name to get individual components
        String[] parts = fullTableName.split("\\.");
        if (parts.length != 3 && parts.length != 4) {
            throw new IllegalArgumentException("Invalid FullTableName format. Expected format: database.schema.table");
        }
        String database = parts[0];
        String schemaName = parts[1];
        String tableName = parts[2];
        String recordId = request.getRecordId();

        // Retrieve OIDs using the getOIDs method
        Map oidMap = getOIDs(networkId, database, schemaName, tableName);
        
        if (!oidMap.containsKey("databaseOid") || !oidMap.containsKey("tableOid")) {
            throw new RuntimeException("Failed to retrieve necessary OIDs for FullClassAKey generation.");
        }

        String databaseOid = Long.toHexString(oidMap.get("databaseOid"));
        String schemaOid = Long.toHexString(oidMap.get("schemaOid")); // Use table OID as a substitute for schema
        String tableOid = Long.toHexString(oidMap.get("tableOid")); // Table OID
        if (recordId != null) {
        	Long.toHexString(oidMap.get("recordOid"));
        }

        // Validate inputs
        if (networkId.length() != 7 || !networkId.matches("[0-9A-Fa-f]+")) {
            throw new IllegalArgumentException("Invalid networkId. Must be a 7-character hexadecimal value.");
        }
        if (memberOrClassId.toString().length() != 7 || !memberOrClassId.toString().matches("[0-9A-Fa-f]+")) {
            throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 7-character hexadecimal value.");
        }

        // Generate the prefix for the ID (Modify to include OIDs)
        byte[] prefix = generateIdPrefix(FULL_CLASS_A_KEY, networkId, memberOrClassId, databaseOid, schemaOid, tableOid);

        // Generate the concatenated hashes based on the prefix (Modify to hash OIDs as well)
        IdTypeBitLengths bitLengths = getBitLengthsForIdType(FULL_CLASS_A_KEY);
        if (bitLengths == null) {
            throw new IllegalArgumentException("Unknown or unsupported idType.");
        }
        TreeMap hashResult = generateConcatenatedHashesWithPrefix(prefix, bitLengths, databaseOid, schemaOid, tableOid, recordOid);

        // Combine the hash sections into a single ID
        StringBuilder fullId = new StringBuilder();
        hashResult.forEach((index, value) -> fullId.append("-").append(value));

        return fullId.toString();
    }

	
	// Method to generate a new Full Class_UR ID and return the result
		public static String createNewFullURId(IdType FULL_CLASS_UR_KEY, String networkId, MemberOrClassId memberOrClassId2, GeneralMessage request) throws Exception {
		    
			// Validate inputs
		    if (networkId.length() != 7 || !networkId.matches("[0-9A-Fa-f]+")) {
		        throw new IllegalArgumentException("Invalid networkId. Must be a 7-character hexadecimal value.");
		    }
		    if (memberOrClassId2.toString().length() != 7 || !memberOrClassId2.toString().matches("[0-9A-Fa-f]+")) {
		        throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 7-character hexadecimal value.");
		    }

		    // Extract database, schema, and table name from request
		    String fullTableName = request.getFullTableName();
		    String[] parts = fullTableName.split("\\.");
		    if (parts.length != 3) {
		        throw new IllegalArgumentException("Invalid FullTableName format. Expected format: database.schema.table");
		    }
		    String database = parts[0];
		    String schemaName = parts[1];
		    String tableName = parts[2];
		    request.getRecordId();

		    // Retrieve OIDs using the getOIDs method
		    Map oidMap = getOIDs(networkId, database, schemaName, tableName);

		    if (!oidMap.containsKey("databaseOid") || !oidMap.containsKey("tableOid")) {
		        throw new RuntimeException("Failed to retrieve necessary OIDs for FullClassAKey generation.");
		    }

		    String databaseOid = Long.toHexString(oidMap.get("databaseOid"));
		    String schemaOid = Long.toHexString(oidMap.get("schemaOid"));
		    String tableOid = Long.toHexString(oidMap.get("tableOid"));
		    String recordOid = Long.toHexString(oidMap.get("recordOid"));

		    // Generate the prefix for the ID
		    byte[] prefix = generateIdPrefix(FULL_CLASS_UR_KEY, networkId, memberOrClassId2, databaseOid, schemaOid, tableOid);

		    // Generate the concatenated hashes based on the prefix
		    IdTypeBitLengths bitLengths = getBitLengthsForIdType(FULL_CLASS_UR_KEY);
		    if (bitLengths == null) {
		        throw new IllegalArgumentException("Unknown or unsupported idType.");
		    }
		    TreeMap hashResult = generateConcatenatedHashesWithPrefix(prefix, bitLengths, databaseOid, schemaOid, tableOid, recordOid);

		    // Combine the hash sections into a single ID
		    StringBuilder fullId = new StringBuilder();
		    hashResult.forEach((index, value) -> fullId.append("-").append(value));

		    String fullClassURId = fullId.toString();

		    // Store the suffix (last part after the last '-')
		    String[] idParts = fullClassURId.split("-");
		    String fullClassURIdSuffix = idParts[idParts.length - 1];

		    // Store in Redis/etcd for later retrieval
		    String cacheKey = "DHT_TX_TRACE_SUFFIX:" + request.getFullTableName();
		    
		    putToEtcd(cacheKey, fullClassURIdSuffix.getBytes(StandardCharsets.UTF_8));

		    return fullClassURId;
		    
		}
		
		public static String createAndStoreFullUId(IdType fullClassUKey, String networkId, MemberOrClassId memberOrClassId, GeneralMessage request) throws Exception {
		    String fullClassUId = createNewFullUId(fullClassUKey, networkId, memberOrClassId, request);
		    
		    // Store the generated FullClassUId in Redis/etcd so that getUseCaseMetadataId can retrieve it
		    String useCaseKey = "tableToUseCase:" + request.getFullTableName();
		    
		    putToEtcd(useCaseKey, fullClassUId.getBytes(StandardCharsets.UTF_8));
		    
		    return fullClassUId;
		}

		
		// Method to generate a new Full Class_U ID and return the result
		public static String createNewFullUId(IdType fullClassUKey, String networkId, MemberOrClassId memberOrClassId, GeneralMessage request) throws Exception {
		    String memberOrClassId3 = memberOrClassId.toString();
	        // Validate inputs
	        if (networkId.length() != 7 || !networkId.matches("[0-9A-Fa-f]+")) {
	            throw new IllegalArgumentException("Invalid networkId. Must be a 7-character hexadecimal value.");
	        } else {
	            if (memberOrClassId3.length() != 7 || !memberOrClassId3.matches("[0-9A-Fa-f]+")) {
	                throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 7-character hexadecimal value.");
	            }
	        }
	        
	     // Split the full table name to get individual components
	        String[] parts = fullTableName.split("\\.");
	        if (parts.length != 3 && parts.length != 4) {
	            throw new IllegalArgumentException("Invalid FullTableName format. Expected format: database.schema.table");
	        }
	        String database = parts[0];
	        String schemaName = parts[1];
	        String tableName = parts[2];
	        // Retrieve OIDs using the getOIDs method
	        Map oidMap = getOIDs(networkId, database, schemaName, tableName);
	        
	        if (!oidMap.containsKey("databaseOid") || !oidMap.containsKey("tableOid")) {
	            throw new RuntimeException("Failed to retrieve necessary OIDs for FullClassAKey generation.");
	        }

	        String databaseOid = Long.toHexString(oidMap.get("databaseOid")); // DHT Use Case DataBase OID
	        String schemaOid = Long.toHexString(2200L); // DHT Use Case DataSchema (public)
	        String tableOid = Long.toHexString(oidMap.get("schemaOid")); // DHT Use Case DataTable OID
	        String recordOid = Long.toHexString(oidMap.get("tableOid")); // DHT Use Case Data Record OID
	        
	        // Validate inputs
	        if (networkId.length() != 7 || !networkId.matches("[0-9A-Fa-f]+")) {
	            throw new IllegalArgumentException("Invalid networkId. Must be a 7-character hexadecimal value.");
	        }
	        if (memberOrClassId3.length() != 7 || !memberOrClassId3.matches("[0-9A-Fa-f]+")) {
	            throw new IllegalArgumentException("Invalid memberOrClassId. Must be a 7-character hexadecimal value.");
	        }


	        // Generate the prefix for the ID
	        byte[] prefix = generateIdPrefix(idType, networkId, memberOrClassId, databaseOid, schemaOid, tableOid);

	        // Generate the concatenated hashes based on the prefix
	        IdTypeBitLengths bitLengths = getBitLengthsForIdType(idType);
	        if (bitLengths == null) {
	            throw new IllegalArgumentException("Unknown or unsupported idType.");
	        }
	        TreeMap hashResult = generateConcatenatedHashesWithPrefix(prefix, bitLengths, databaseOid, schemaOid, tableOid, recordOid);

	        // Combine the hash sections into a single ID
	        StringBuilder fullId = new StringBuilder();
	        hashResult.forEach((index, value) -> fullId.append("-").append(value));

	        return fullId.toString();
		    
		}

		public static String getRecordId() {
			return recordId;
		}

		public static void setRecordId(String recordId) {
			IdManager.recordId = recordId;
		}


		public static byte[] decode(String nodeId) {
		    if (nodeId == null || nodeId.length() % 2 != 0) {
		        throw new IllegalArgumentException("Invalid hex string: " + nodeId);
		    }

		    int len = nodeId.length();
		    byte[] data = new byte[len / 2];

		    for (int i = 0; i < len; i += 2) {
		        data[i / 2] = (byte) ((Character.digit(nodeId.charAt(i), 16) << 4)
		                            + Character.digit(nodeId.charAt(i+1), 16));
		    }

		    return data;
		}

		private static final HexFormat HEX = HexFormat.of();

		/** Resolves a QUIC endpoint by nodeId hex string (e.g. 0x00A123...) */
		public static String resolveQuicEndpoint(String nodeIdHex) {
			return "quic://" + nodeIdHex + ".quic-network.svc.cluster.local";
		}

		public static String createAndStoreFullUId(IdType type, Cluster networkId, MemberOrClassId memberOrClassId, GeneralMessage request) {
			String data = networkId.getId7() + memberOrClassId.getIdentifier()
					+ request.getFullTableName();
			return hashWithPrefix(type, data);
		}

		public static String createNewFullURId(IdType type, Cluster networkId, MemberOrClassId memberOrClassId, GeneralMessage request) {
			String data = networkId.getId7() + memberOrClassId.getIdentifier()
					+ request.getFullTableName() + request.getRecordId();
			return hashWithPrefix(type, data);
		}

		public static String createNewFullAId(IdType type, Cluster networkId, MemberOrClassId memberOrClassId, GeneralMessage request) {
			String data = networkId.getId7() + memberOrClassId.getIdentifier()
					+ request.getFullTableName() + request.getRecordId();
			return hashWithPrefix(type, data);
		}


		/** SHA3-512 hashing with prefix from IdType */
		private static String hashWithPrefix(IdType type, String data) {
			byte[] sha = new SHA3.Digest512().digest(data.getBytes(StandardCharsets.UTF_8));
			String suffix = HEX.formatHex(sha).substring(2); // keep 510 bits after 2-char prefix
			return type.getHexPrefix() + suffix;
		}
		}


		// ✅ Static reverse lookup table
	private static final Map BY_PREFIX = new TreeMap<>();

	static {
		for (IdType type : IdType.values()) {
			BY_PREFIX.put(type.getPrefixByte(), type);
		}
	}

	public static IdType fromPrefixByte(byte prefixByte) {
		return BY_PREFIX.get(prefixByte);
	}


}

~~~~~~~~~~~~~~~~~~~~~~~

Routing Queries and Locating Nodes

without relying on the Domain Name System

Kademlia Distributed Hash Table (DHT) Protocol

The Kademlia Protocol is the (Unsecured) Foundation for Carrier 2

Kademlia can use UDP, TCP or other protocols for system communication
The Trinity-Tech team has implemented a Server Node as a Java Supernode, on TCP. We will upgrade to QUIC (Quick UDP Internet Connections) using TLS 1.3 and HTTP 3.
ITOTCCA is extending this with a Consensus-Based, Replicating Database/Middleware Distributed System (BFT-SMaRt-on-PostGIS) which is subjected to a 29 Stage Data Integrity Test for critical transactions
Trinity-Tech & ITOTCCA added security mechanisms – TLS encryption, Authentication with Ionic, Kubernetes and PostGIS RBAC (Authorisation mechanics)

Kademlia: Petar Maymounkov & David Mazieres, New York University, 2002

Structure (from Wikipedia)

“The structure of a DHT can be decomposed into several main components.[13][14] The foundation is an abstract keyspace, such as the set of 160-bit strings. A keyspace partitioning scheme splits ownership of this keyspace among the participating nodes. An overlay network then connects the nodes, allowing them to find the owner of any given key in the keyspace.

Once these components are in place, a typical use of the DHT for storage and retrieval might proceed as follows. Suppose the keyspace is the set of 160-bit strings. To index a file with given filename and data in the DHT, the SHA-1 hash of filename is generated, producing a 160-bit key k, and a message put(k, data) is sent to any node participating in the DHT. The message is forwarded from node to node through the overlay network until it reaches the single node responsible for key k as specified by the keyspace partitioning. That node then stores the key and the data. Any other client can then retrieve the contents of the file by again hashing filename to produce k and asking any DHT node to find the data associated with k with a message get(k). The message will again be routed through the overlay to the node responsible for k, which will reply with the stored data.”

So, clearly, in the case of providing Database-Node Access, our task is to utilise adapted ‘put(k, data)’ and ‘get(k)’ messages to access Database Records which are effectively the file-data mentioned above. This would mean not only assigning a filename (using a [network, database, schema, table, record-id] identifier) to every database record across clusters of networks and databases, but also assigning nodes to individual “client devices”. So a record identifier [n, d, s, t, r] would be SHA-1 hashed to produce the 160-bit key (k), and this set of keys would exist for all nodes (ie client devices and database records). Each client device would participate in 2 separate systems

A Peer to Peer network (P2P), utilising Etcd for local data storage on each node, and,
The Enterprise Database Access Network (EDAN), based on PostGIS Database Nodes residing on Elastos Carrier SuperNodes, and incorporating a Business Peer to Peer Network, separated from the general P2P Network, which uses a separate partition on etcd.

On the EDAN there would be Client and Server Nodes, with each Database Record [n, d, s, t, r] representing a “filename” (replicated) on all Server Nodes in a Business Network, and each Client Device representing one Client Node [client’s address on the Elastos Network].

The EDAN will be sharded into mutually exclusive Partitions comprising one Business Network each, under their own Supernodes. So the keys would apply independently for each business network, including only its own Client and Server Nodes, with each
Database Record N:{d}:{s}:{t}:{r}
representing one traditional Kademlia “filename”, replicated on all Servers (Nodes) on Network N, and each Client Device representing one Client Node [the client’s address really just an Id on their own Enterprise Network].

Partly due to the real threat to encryption from standard computational capacities, which have surpassed the strength of 160-bit keys, and partly due to the advanced threat from Quantum Computers, we have decided to increase the 160-bit keys used by a traditional Kademlia Network, to a SHA-3 Encrypted 512 bit key. The system is shown above. Our vision is of one Enterprise Network per Supernode, which will be ostensibly isolated, although restricted communication will be possible (and necessary) between Supernodes. Such Business Networks would typically be comprised of up to several dozen “Member Classes” (although there is no practical upper limit). Please refer to The General page for an explanation of the concept of Member Classes, or look at The Medical or CHEIRRS pages for practical examples. In general terms one Member Class corresponds to one database Node, where all nodes in any Business Network will act as Master copies – carrying copies of the entire Business Network’s databases (with one database per Member Class). The data in each database is client-side encrypted to ensure privacy.

A robust permissions and authorisation system operates within each Member Class database and within each participating Enterprise schema. If appropriate (due to the network’s structure) or simply when demanded, a Member Class node may be occupied by one enterprise alone – ie by one Member. Also it should be understood that in the following code, bootstrapping refers to normal “Secondary” nodes, as opposed to the broader process of bootstrapping a SuperNode or a Network of SuperNodes. It is also necessary to incorporate a process of Registering (ie bootstrapping to Secondary nodes) client nodes or personal devices (“Tertiary” Nodes). The former is one actual Task defined for the code repository at the head of this page (ie we are developing a Database Network Bootstrap method), however for this to work, there is a need for Bootstrapping SuperNodes to exist and be running. Therefore the Initialisation and Startup of an inaugural Elastos.Carrier.Java SuperNode is first required.

The process involving the Transaction Traces incorporates a recording process on the etcd DataStores on each DHT (Kademlia) node (separate to the Enterprise PostGIS databases), where the 17-data-field Traces are intercepted at the point of receipt, and added as “leaves” to a Merkle Tree formed by these transaction blocks’ traces. (All trace records are archived to the local Supernode’s PostGIS database after each Slab is completely processed.) After all traces, in ‘Slabs’ of 24 Blocks of 1024 transactions per Block, have been added, the Merkle Root of the Slab’s Tree is calculated, recorded on the Elastos Blockchain, and a comparison is conducted between evidence of the previous Root retrieved from the Elastos Blockchain, with that obtained by extracting the previous Root from the current Database State, since the value is also stored on the Metadata DataStore . We then rebuild the Merkle Tree for the previous slab using the transaction trace database’s history and compare the rebuilt root with the one retrieved from the blockchain. If the rebuilt root differs from the blockchain root, it could indicate tampering in the underlying transaction data. This approach verifies not only the stored root but also the integrity of the transactions that contributed to it. We validate that every transaction contributing to the current Merkle Tree has a corresponding entry in the metadata store. We ensure that these entries match the expected data (e.g. transaction IDs, hash values). Instead of relying solely on the Merkle Root, we include “hashes” of individual transactions in the leaves of the Merkle Tree.

This ensures that even if the Merkle Root is consistent, individual tampered transactions can be identified by checking their hashes. Any mismatch between the transaction hash in the database and the blockchain indicates tampering.
We use periodic snapshots of the database, stored securely in an independent location, to cross-validate the database state. Snapshots are used to detect discrepancies that might otherwise go unnoticed due to tampering with live data. Thus we take database snapshots at the end of each slab. These snapshots are securely stored using immutable storage. We use the snapshots to rebuild and validate historical Merkle Trees.
Proof-of-Consistency algorithms (like those used in blockchain systems) ensure that:

The current Merkle Root is consistent with the previously recorded root.

No invalid transactions have been added between slabs.

Any discrepancies arising in any of these procedures indicate a tampered or otherwise faulty intervening process, including possibly corrupt database transactions. Such a result would be notified automatically to all stakeholders, and would be followed by a full Root Cause Analysis.

As part of a sustained effort to eliminate reliance on DNS, IP Addressing and the use of standard TCP Ports, we have commenced redevelopment with QUIC (Quick UDP-based Internet Connections). QUIC uses TLS 1.3 on HTTP3. Together with gRPC, the combination yields fast, low-latency, error-corrected connections with increased security and is compatible with our Kademlia DHT network and (with the help of etcd distributed storage) does allow for Peer Discovery and Connection without DNS. Pure embedded DHT. We are currently removing all dependencies relating to DNS, IP Addressing and TCP Ports across all our components, with the intention of removing the dependency for transport in BFT-SMaRt on Java’s Netty Server and Client. Similarly we are progressing to eliminate reliance on the Carrier 2 section of the system (ie the DHT section). We are moving to a jvm-libp2p “backbone”.

Steps to Transition from TLS 1.2 to QUIC (with TLS 1.3)

Replace Transport Layer
1. Identify parts of BFT-SMaRt using Netty-based TLS.
2. Integrate a QUIC library into these layers to replace the existing TCP+TLS 1.2 stack.
Update Communication Protocols:
1. Update gRPC services to use QUIC endpoints.
2. Define gRPC services over QUIC (e.g., quic:// endpoints) and test their compatibility with existing components.
Modify Replica Communication:
1. Replace internal socket communication (or InetSocketAddress) with QUIC endpoints.
2. Ensure the replica state synchronization mechanisms support the new transport layer.
Integrate with Existing Security Frameworks:
1. Generate TLS 1.3-compatible certificates using Cert-Manager or equivalent.
2. Use QUIC libraries supporting certificate management, such as quiche or ngtcp2.
Test and Validate:
1. Conduct interoperability tests between nodes using QUIC and TLS 1.3.
2. Validate consensus mechanisms with the updated transport stack.
Deploy Gradually:
1. Use a hybrid setup where replicas communicate over both QUIC (TLS 1.3) and the legacy TLS 1.2 transport, ensuring compatibility during transition.

How QUIC Replaces Netty

Netty servers are the core transport layer for BFT-SMaRt’s communication. Transitioning to QUIC involves:

Replacing Netty with a QUIC Implementation: A QUIC library such as quiche or Google’s QUICHE replaces Netty for transport.
Adapting the Protocol Stack: QUIC operates over UDP instead of TCP, so the BFT-SMaRt communication stack needs to adapt to this difference.
gRPC Over QUIC: Since we are using gRPC for application-level communication, transitioning to gRPC over QUIC simplifies the transport layer adaptation. ie redirect all gRPC calls in the application layer to use gRPC over QUIC
Implement the necessary QUIC endpoints and integrate them into the Replica Communication Logic

Advantages of Replacing Netty with QUIC

Improved Performance: QUIC reduces latency through 0-RTT (Zero Round-Trip Time) and connection multiplexing. Security Enhancements: TLS 1.3 is built into QUIC, eliminating the need for separate TLS configurations. Better Suitability for BFT Systems: QUIC’s resilience to packet loss and faster recovery can benefit BFT-SMaRt’s performance under high churn.

Challenges

Library Compatibility: Ensure the QUIC library you choose supports Java and provides a stable API for integration.
Code Refactoring: Significant changes to the communication layer of BFT-SMaRt will be required to replace Netty with QUIC.
Testing: Extensive testing is needed to validate that QUIC provides the expected performance and reliability improvements.

Gradual Replacement Strategy

To ensure a smooth transition:

Implement a Hybrid Approach:
1. Use QUIC for new communication pathways (e.g., gRPC services).
2. Retain Netty for existing BFT-SMaRt communication temporarily.
Develop QUIC-Based Communication in Parallel:
1. Implement QUIC-based endpoints for replica communication.
2. Test these endpoints in isolation before full integration.
Migrate Fully to QUIC:
1. Once all communication pathways are tested and stable, deprecate Netty-based transport.
2. Remove Netty dependencies from the codebase.

QUIC Integration Workflow

Replace Netty Sockets: Replace all instances of NettyServer and NettyClient in BFT-SMaRt with equivalent QUIC-based implementations.
Update Configuration Files: Add QUIC-specific configurations (e.g., certificates for TLS 1.3, UDP settings).
Test Performance: Evaluate performance under different network conditions and compare with Netty.

BFT-SMaRt will leverage the advantages of QUIC while maintaining its robust BFT capabilities.

By incorporating QUIC, we intend to modernize BFT-SMaRt’s security model and gain the performance

benefits and flexibility that come with TLS 1.3 and QUIC’s efficient transport mechanisms.

The following code is of a general nature, not meant to correspond precisely with actual Carrier 2 code.

Kademlia Node Initialization

java

Client Node Bootstrapping
java

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.nio.charset.StandardCharsets;
import java.util.List;

public class ClientNode {
     private String nodeId;
     private String networkId;
     private RoutingTable routingTable;
     private KademliaNetwork kademliaNetwork;

     public ClientNode(String networkId, List bootstrapNodes) {
       this.networkId = networkId;
       this.nodeId = generateNodeId();
       this.routingTable = new RoutingTable();
       this.kademliaNetwork = new KademliaNetworkImpl(networkId, nodeId, routingTable);

       // Bootstrap the node
       bootstrapNode(bootstrapNodes);
     }

     private String generateNodeId() {
       try {
         MessageDigest digest = MessageDigest.getInstance(“SHA-3”);
         byte[] hashBytes = digest.digest(Long.toString(System.nanoTime()).getBytes(StandardCharsets.UTF_8));
         StringBuilder nodeId = new StringBuilder();
         for (byte b : hashBytes) {
           nodeId.append(String.format(“%02x”, b));
         }
         return nodeId.toString();
         } catch (NoSuchAlgorithmException e) {
         throw new RuntimeException(“SHA-3 algorithm not found”, e);
       }
     }

     private void bootstrapNode(List bootstrapNodes) {
       for (String bootstrapNode : bootstrapNodes) {
         // Add bootstrap node to routing table
         routingTable.addNode(bootstrapNode);

         // Perform initial lookup to populate routing table
         performInitialLookup(bootstrapNode);
       }
     }

     private void performInitialLookup(String bootstrapNode) {
       // Simulate a lookup operation to populate the routing table
       List contacts = kademliaNetwork.lookupNode(bootstrapNode);
       for (String contact : contacts) {
         routingTable.addNode(contact);
       }
     }

     public static void main(String[] args) {
       String networkId = “N1”;
       List bootstrapNodes = List.of(“bootstrapNode1”, “bootstrapNode2”);

       ClientNode node = new ClientNode(networkId, bootstrapNodes);
       System.out.println(“Node ID: ” + node.nodeId);
       System.out.println(“Network ID: ” + node.networkId);
       System.out.println(“Routing Table: ” + node.routingTable);
     }
}

Summary
We must uncover or create the corresponding Real Classes in Carrier 2, that do the following:

Kademlia Node Initialization: Handle the creation of a unique node ID and assigns the node to a shard.
Routing Table Implementation: Manage known nodes for Carrier routing purposes.
Kademlia Network Interface and Implementation: Carrier Network operations for node lookup.
Database Node Bootstrapping: Bootstrap a database node, ensuring it connects to both the Carrier network (and populates its routing table) and the overlaying BFT-SMaRt network.
Client Node Bootstrapping: Bootstrap a client node, ensuring it can interact with both the Carrier and BFT-SMaRt networks for storing metadata, and storing and retrieving enterprise database records.
Composite Key Management:
Use composite keys to ensure data is uniquely identifiable and correctly routed within the shard.
Incorporate Network ID, database name, schema name, table name, and record ID in the composite key.
Cluster Configuration:
Each database cluster contains all databases for the range of member classes in the shard.
Each cluster node is replicated as many times as there are member classes, ensuring redundancy and fault tolerance.
Transaction Management:
BFT-SMaRt will handle transaction ordering and replication across the database nodes, ensuring consistency and reliability.
Shard-Specific Operations:
Direct storage and retrieval operations to the appropriate shard.
Ensure that all database operations are confined to the correct database cluster based on the composite key.
Detailed Implementation
Composite Key Generation and Hashing in Java

java

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.nio.charset.StandardCharsets;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.HashMap;
import java.util.Map;

public class KademliaPostGISExample {

     // Generate composite key including the network ID
     public static String generateCompositeKey(String networkId, String database, String schema, String table, String recordId) {
     return String.format(“%s:%s:%s:%s:%s”, networkId, database, schema, table, recordId);
   }

   // Hash composite key using SHA-3
   public static String hashCompositeKey(String compositeKey) {
     try {
       MessageDigest digest = MessageDigest.getInstance(“SHA-3”);
       byte[] hashBytes = digest.digest(compositeKey.getBytes(StandardCharsets.UTF_8));
       StringBuilder hashedKey = new StringBuilder();
       for (byte b : hashBytes) {
         hashedKey.append(String.format(“%02x”, b));
     }
     return hashedKey.toString();
       } catch (NoSuchAlgorithmException e) {
       throw new RuntimeException(“SHA-3 algorithm not found”, e);
     }
   }

     // Store record in PostGIS database, considering the shard
     public static void storeRecord(String networkId, String database, String schema, String table, String recordId, Map<String, Object> recordData) throws Exception {
       String compositeKey = generateCompositeKey(networkId, database, schema, table, recordId);
       String hashedKey = hashCompositeKey(compositeKey);

     // Connect to the PostGIS database cluster for the specific shard
     try (Connection conn = DriverManager.getConnection(“jdbc:postgresql://localhost:5432/” + networkId, “user”, “password”)) {
       String sql = “INSERT INTO records (key, data) VALUES (?, ?::jsonb)”;
       try (PreparedStatement stmt = conn.prepareStatement(sql)) {
         stmt.setString(1, hashedKey);
         stmt.setString(2, recordData.toString()); // Assuming recordData is JSON serializable
         stmt.executeUpdate();
       }
     }
   }

     // Retrieve record from PostGIS database, considering the shard
     public static Map<String, Object> retrieveRecord(String networkId, String database, String schema, String table, String recordId) throws Exception {
       String compositeKey = generateCompositeKey(networkId, database, schema, table, recordId);
       String hashedKey = hashCompositeKey(compositeKey);

     // Connect to the PostGIS database cluster for the specific shard
     try (Connection conn = DriverManager.getConnection(“jdbc:postgresql://localhost:5432/” + networkId, “user”, “password”)) {
       String sql = “SELECT data FROM records WHERE key = ?”;
       try (PreparedStatement stmt = conn.prepareStatement(sql)) {
         stmt.setString(1, hashedKey);
         try (ResultSet rs = stmt.executeQuery()) {
         if (rs.next()) {
         // Assuming the data is stored as JSON
         return new ObjectMapper().readValue(rs.getString(“data”), Map.class);
         }
       }
     }
   }
   return null;
}

     public static void main(String[] args) throws Exception {
       String networkId = “N1”;
       String database = “db1”;
       String schema = “schema1”;
       String table = “table1”;
       String recordId = “12345”;
       Map<String, Object> recordData = new HashMap<>();
       recordData.put(“name”, “Alice”);
       recordData.put(“age”, 30);

// Store the record
storeRecord(networkId, database, schema, table, recordId, recordData);

       // Retrieve the record
       Map<String, Object> retrievedData = retrieveRecord(networkId, database, schema, table, recordId);
       System.out.println(retrievedData);
     }
}

~~~~~~~~~~~~~~~~~~~~~~~

DISCLAIMER:

The sense in which the terms “Automated Trust”, 100% FID, etc are used here should be taken to indicate that Trust in the Owners and Operators of a Database System can be guaranteed, given that the initial Coding and Development process is Transparent to all Business Customers of the developers/SaaS (Software as a Service) providers. This is because the Business Customers (with sound Governance) actually Operate their own Member Class servers under leases from the SaaS providers. As operators, they are expected to look after their own IT interests, but they are not expected to need to Trust any Operators in the system of Database server nodes. The coded system enforces Trustworthiness in a strict way, and programmatically, amongst all operational parties. The old style of Trusting one or several Systems Administrators as Superusers with ultimate control can be over. There does, admittedly, still need to be an ultimate owner or Superuser at the root of the “tree”, however this role will be taken by individual Directors of ITOTCCA, and would only be enacted on the rare occasions of major alterations to the structure and functioning of the Database System. Almost all operations will be handled by the Database Administrators, sub-contracted under the auspices of each Member-Class consortium, with a minimum set of powers to achieve their duties effectively. It would obviously be against the interests of any Director of ITOTCCA, to participate in Fraud or Embezzlement. The culture at ITOTCCA is open, honest and transparent, but watchful.

Nevertheless, the committing of a fraudulent act by a corps of over 33% of the Member Class “sites” in a Network, acting together, is possible. Such an act would be notified automatically to all stakeholders, including all Member companies of all affected Member “Classes”, immediately upon the next consistency check against the Elastos Blockchain, scheduled to occur at every 24 Block committals. PostGIS will have been programmed to record traces continuously, forming an ultimately reliable audit trail, which can be compared against the databases by starting from the last clean block on the database and obtaining logs of the Database Operations from that point forward, as recorded on PostGIS and ultimately as Merkle Roots on the Elastos/Ethereum blockchain when analysis of transaction traces is completed, after each of 24 block commitals. If the notified lists of executed and committed (or rolled-back) transactions (together with their Block Orderings) are not internally consistent, as well as consistent when cross-compared with the separately intercepted and pre-registered lists of originally submitted transactions, and also consistent with the previous sequences of Merkle Roots on the blockchain, an Alarm is Multicast to all Stakeholders, and the SaaS provider’s Contingency Plan is implemented immediately. The SaaS provider’s own Governance obviously requires a sound and complete Contingency Plan for such Breaches, extending to specifying Member Companies’ contractual duties to have and understand their own External Data Inconsistency Contingency Plan. It is also necessary to detect false positive indications before taking further action.

In all cases of an Alarm being multicast, a Root Cause Analysis is required, where the SaaS provider is the responsible party, in consultation with all affected Members.

However, some Fraudulent acts committed by employees of Member companies, cannot be prevented by the SaaS provider, nor detected (in the absence of an Audit of the Company’s Accounts). That is to say, the responsibility for the majority of Trust situations with respect to Member company employees, falls to the employer, not the SaaS provider. This is because it is quite possible to present an appearance of externally consistent and Trustworthy transactions while indulging in Fraud, embezzlement or exfiltration of data. External Consistency is not the same as Internal Consistency. Internal Auditing and a sound Culture at the company can assist. In addition, a comprehensive actvity monitoring system, led automatically by Artificial Intelligence is possible, and would all-but seal system security. One example of a system such as this, offered by ITOTCCA, is the ELK Stack (Elasticsearch, Logstash, Kibana) with Machine Learning. In general the contribution that the SaaS provider can make here is by providing the immutable and non-repudiable (user-signed) Blockchain and Database logs and records. All transactions (and not just financial ones) are accompanied by the Digital signature of the client (person), or at least of a device owned, and registered by them on the Elastos System. If agreed, we can provide access to the ‘Elastic’ ELK Stack for a low monthly price. Basically the Elastic Stack employs Artificial Intelligence, training as it works, to drive an Anomaly Detection System covering user and machine behaviour.

In summary we are guaranteeing “External Consistency”, ie external to Member companies, and by adding the ELK Stack we can extend that guarantee to your entire system. The exception to these guarantees is in extremely rare cases where False Negatives are encountered, that is, when no concerns are reported, despite the existence, in reality, of concerns. We would expect to be sued over such occurrences, and we have Product Liability Insurance in place. That is not to say that we expect such an event to occur.

—

Front End (GUI) applications come “seeded” with the Identities of available bootstrapping nodes for authenticated clients’ business networks. The Kademlia Protocol allows the client node to discover the peer server which registers and Bootstraps the client. The same peer discovery procedure is followed in subsequent sessions. Bootstrapping occurs once only per client device.

So there is metadata in the format of JSON fields that form the basis of selections from which queries are constructed, indexed by DHT network record keys. The second type of metadata is the transaction traces, which are generated by the latter database queries and which are used by ChubbyChecker and the supernode network’s SuperChecker, to perform the 24-item trace-cross-checks to validate the integrity of the Enterprise (business data) and Supernodes (metadata) Databases.

An important role in the trace metadata processing is to calculate and store transaction trace Merkle Leaves in preparation for writing calculated Merkle Roots to the Elastos ELA/ESC Sidechain. This anchors an immutable Audit Trail to the Blockchain. At every occasion of writing a new Root, the true previous root is compared to database evidence of the previous root to check consistency, before writing the current root value. For both the JSON-encoded database query metadata and the trace metadata, a PostGIS Archival metadata database, as Storage. The secondary (Worker) nodes push (synchronise) metadata regularly to the supernode databases. From the supernode network, as our Control Databse Subscribes to a Message Broker, in turn subsribing to the Supernodes, then republishes the messages (data), and our Control Database collects those archives for analysis (by ChubbyChecker) and permanent storage.

The system we employ for Database Access involves an indexed approach to the PostGIS Enterprise databases, which operate via the Kademlia DHT network. The system enables the DHT network to use a PostGIS Enterprise Database record’s Id or “key” as a pointer to the corresponding Trace Database Record. The 2 records have identical primary keys (on the separate PostGIS databases).

Overall, this has several benefits (aside from the anti-hacking benefit of avoiding DNS) such as improved fault-tolerance and scalability, and looser coupling of systems, but does have the drawback of increased latency. Whilst this can be mitigated to a certain extent, we view increased latency as the price of security.

We are developing the final forms for the supernode and secondary nodes’ operational (lifecycle) loops. We are designing the loops to work with BFT-SMaRt and the DHT networks and operations decoupled. In this fashion the full benefits of each of these twin networks will be realised.

We believe that the safest strategy for privacy and security in the cases where data, to be processed, must be decrypted in order for operations to be possible, is to organise for the client to use our public key to encrypt data (meaning only ITOTCCA can decrypt it), then for the data to be operated upon as decrypted data, and the results encrypted by the client’s public key (so that only the client can decrypt it) and stored on the client’s database.

The secondary (Worker) nodes also require Bootstrapping by a pre-existing node. Supernodes take this role. The tertiary (Client) nodes have node Ids from the same 512-bit keyspace as all Identifiable components of the DHT system, including Data Records, Value Keys (Value Ids), and Node Ids.

The following image is a representation of one node (of 13) in the recent state of our development platform. We are now heavily influenced by the prospect of automated trust being built-into a database multi-site installation. It appears it is possible to make a Postgres database behave with equal integrity and immutability to a Blockchain, by following certain development procedures, which have been published as Plain English (ie no code was published) research method summaries in a paper by IBM India (see below).

This image represents one node in the developing Automated Trust System. It is fully Industrial-strength and IoT-ready. The following image leaves out the pan-database-server Bucordo Ordering/Replication system.

kubeeverything

The IBM paper, originating in IBM India, https://arxiv.org/pdf/1903.01919.pdf is linked to the following images. It is the inspiration for our efforts to provide an Ultimately Trustworthy Installation, in the cloud, but here shown representing the present stage of development of a multi-node assemblage using Kubernetes in the Cloud. Thus, by following the IBM research lead, we can say that in production, there would generally be one copy of the above installation per member-class, and arranged, configured and coded to ensure these databases interact as Masters in a data-intensive Blockchain-type assemblage. Therefore, in our working k3d (k3s in Docker) development model, there are 14 (2 clusters of 7 agents)for das_fuhrwerk, plus 13 of the internetworked member classes as may exist on their own Sample Business Subnet (ideally, alongside other member-classes from other Business Subnets in production) e.g. member-class(n,m) with n == business sub-network id and m == member-class id. Within member classes there may be many (or few) actual independent members (Companies). The following images detail the Bucordo Ordering & multi-Master Replication system and the Node-Red-Industrial system, in its current development phase.

We actually have a range of 13 Member Class “sites” (plus “das_fuhrwerk representing our company) implemented on a k3d setup, each connected to a development-ready, Pg-Admin4-administered, PostgreSQL replica, of the master database.

We are, finally, using the k3d assemblage to develop the BFT-SMaRt Client/Server Distributed Applications we have been planning for some time. The future is looking bright. We are receiving some assistance with the development of the base Front Ends (in Typescript), built around the Elastos.Essentials (github) package. Development of the Front End is progressing well.

At the currrent stage, we are looking at diving into the Postgres Source Code in order to begin development of the parts of the Ordering and Replication system required to complement BFT-SMaRt operations, and so that we can effectively implement the code changes specified in the IBM article.

Bucordo Merges with the Elastos Carrier SuperNode Project to Produce a Database-Node Tier, that requires the original and primary Java-coded SuperNode Tier, in order to be bootstrapped onto the Distributed Hash Table (DHT) networked system.

This project was based on Bucordo (on k3d Kubernetes); merged with Cyber Republic’s Elastos Carrier 2 (gRPC- & REST-based) Blockchain and Net Transport Technology. Carrier 2 is a 21st Century Distributed Node Communication Network, based on the Kademlia Protocol, allowing the Domain Name System to be bypassed for non-traditional (Web 3) networking requirements (including Enterprise Database access), yet enabling traditional Web2 technologies (traditional website access, email, etc) to work. The blockchain end of a transaction will occur first, followed by writing or reading of data to or from PostGIS. Our intention is to enhance upon the lead provided by research from IBM in India, where a practical way of making a database as immutable and tamper-proof as a blockchain, has been discovered, for the purposes of Smart Contract Trading and any internetworked transactions. The paper, “Blockchain Meets Database: Design and Implementation of a Blockchain Relational Database” ∗ Senthil Nathan 1 , Chander Govindarajan 1 , Adarsh Saraf 1 , Manish Sethi 2 , and Praveen Jayachandran 1 1 1 IBM Research India, 2 IBM Industry Platforms USA (snatara7, chandg12, adasaraf, praveen.j)@in.ibm.com, 2 manish.sethi1@ibm.com

.. refer to this link: https://arxiv.org/pdf/1903.01919.pdf

.. or click on the previous diagram ..

has revealed that it is possible to provide Trust in a natural way in networking between Companies, whilst basically distrusting each other, by making a network of PostGIS Databases behave like a large-data-capacity blockchain. To ITOTCCA, this is revolutionary. Prospective customers will no longer have to trust our company to any extent as far as daily operational security is concerned. Neither do they have to trust other companies. The idea is to allow the machines operated by each Company-Class to keep each other honest, and to automatically report security exceptions to all Stakeholders including our Board and all member companies affected. The role played by Carrier2 concerns the prevention of typical Hacking attack modes such as Device Masquerading, Man in the Middle and Denial of Service Attacks. The use of Client-Side Encryption and regular backups stored offline, subverts attempts at Ransomware attacks. Regular updates to Unix-based servers constitute more effective security for the virtual machines, including protection against Malware, while RBAC (Role Based Access Control) lists ensure proper Authorisation practice. Authentication is via identities registered and recorded on the Elastos DID Sidechain, a distributed Blockchain based system. We employ 2-factor authentication.

There is a hole in the plan, in that if globally over 33% of Ordering nodes were to be caused to act fraudulently, the factor of Trust would be destroyed. Nevertheless, elsewhere (on our website at itotchaincloud.com.au/security) we note the contribution of Rong Chen and Elastos in devising a hardware enhancement which is necessary for Blockchain Miners to be fitted with in order to be able to participate in Mining. This device ensures by design that no corps can act together in a gang of >33% ownership and/or direction. The idea is currently operational with Elastos. It appears to be possible, therefore, to utilise the Elastos Blockchains themselves to provide an immutable Fraud Check service against the database.

These developments have caused ITOTCCA to review our monolithic design, opting now for an isolation of Business Networks at the encryption/permissions level from each other and restricting the scope of connectivity to be strictly internal to each separate Business Network (of course external web-connections will still be possible with REST APIs, but safely shielded by whitelists under Carrier (see itotchaincloud.com.au/security). Future enterprise growth in network demand requires networks to be performance optimised and segmented (isolated) to provide access with performance. The design is now to have a range of cloud servers per Business Network, usually with one “site” per Member Class (or per Member for Government Departments and large Organisations). The optimum value for the total number of member_class sites per Cluster (possibly containing several Business sub-nets, or on the contrary with one Business Network spanning more than one cluster) is not known at this stage, although the apparent limit here, revealed by developing with BFT-SMaRt, is 12 member-classes for customers + 1 (for our Das_Fuhrwerk schema in each installation).

Each member-class (represented by its own schema and dApp) would ideally take responsibility for its own virtual hardware. This requires a Governance framework to be established in each Business Sub-Net. The virtual servers are all to be linked (within each of the n sub-networks across all installations, between member-class nodes within a Master/Master replication system. All member-classes carry identical copies of the entire installation’s databases, with one strictly private Business Channel per member (company), enforced by Client-Side Encryption in the cloud, in addition to the RBAC (Role Based Access Control) permissions system. Here, the machines and networks are designed with the goal of ensuring clean play at all times, following the blockchain model.

The following series of 5 diagrams may help clarify some details of the Trust System being developed:

The previous diagram represents a “Serialised” approach to Transaction execution, ordering and committal, whilst the following diagram shows a “Parallel” approach, where Transaction Execution, Ordering and Committal occur simultaneously with other stages of the same transaction-ordering processes.

IBM’s initial research reported a minor preference in efficiency for Parallel Processing, however our chosen transaction ordering and replication system appears to work more naturally in a serial mode.

One positive aspect to the use of a BFT-SMaRt distributed network of databases is that this design fits perfectly within a broader network of client nodes that use the Kademlia Distributed Protocol.

In production we intend to operate clusters of nodes (possibly as Supply Networks, Regional Business Networks or as simpler Mutually Secure networks comprised of individual and independent members) composed of 3 containers on each node: 1 PostGIS Database container (C++/PLpgSQL-extended, copyright) + 1 BFT-SMaRt-[reorganised, extended, copyright] container (Java) + 1 ChubbyChecker-[copyright] container (GO), as well as a separate Carrier 2 Supernode, with accompanying SuperChecker, the equivalent of ChubbyChecker, but for the Supernodes, providing overarching but low-level control of the Kademlia Metadata Network.

In the above IBM article, it is envisaged that a section of non-internetworked transactions would be outside the network. This implies that a company’s own employees, and their connecting devices, can be trusted more than the internetworked transactional parties. We believe this to be highly questionable. For example a recent major attack originated when an employee simply synced their work browser with their unsecured home browser.

Enter the Elastos DID Sidechain, Elastos Carrier2 and Elastos Blockchain.

To this end, and in the spirit of the Elimination of Risk where feasible, we are striving to implement a Hybrid model with 2 layers. We intend to offer a system which, for all financial, or otherwise critical, transactions or actions, records traces on the Elastos Blockchain,

This is responsive AI-led production, backed by a secure set of immutable audit trails, and based on Trust paid for by our investment in the Trust Mechanics of Elastos (as we pay our transaction fees), also providing a built-in insurance guarantee against a global fiat currency collapse by using this Elastos system. When you add to that the revolutionary ultra communications security enabled by Rong Chen and his Elastos Carrier system (with the Enterprise strength version 2, as adapted by ITOTCCA), the argument becomes more compelling.

The hole in the would-be iron-clad system, is the factor of trust demanded by the providers and administrators of the Centralised Cloud Database system, in a setup we call Hybrid Web2/3, which computational and network efficiencies simply do mandate. That is to say, current capacities of Networks, Devices & Blockchains are not sufficient to enable storage of Enterprise quantities of Data, On-Chain. We continue to require databases.

Our solution to the question of trust, is to follow the IBM India lead, and develop a mutually assured Trust system involving all database servers, across all business subnetworks i.e. supply chains) in an installation, which is designed to keep all players honest, and which is ultimately guaranteed by our copyright use of Elastos Carrier 2.0 and ELA/ESC Sidechain. Our methods are referred to by the package name “ChubbyChecker“. We use Elastos because it enables us to lay off the relatively high risk of fraud on the database networks (under control of BFT-SMaRt) , onto the very low risk blockchain, by recording the Merkle Roots of the Merkle Trees formed from traces of the transactions, after (1) they are intercepted and recorded as individual transaction traces, at the middleware servers, ie upon submission (Pre-Registration), (2) as they are calculated per Block as Slabs of Blocks are produced, at the middleware servers, and (3) as they are separately compared against the databases at every Block Commit (Post-Registration), onto the chain, so that when the 24th block (one Slab) in a cycle is committed, we cross-check the records on the chain for external and internal consistency, employing 24 cross-check-items, including a comparison of state change hashes across all database nodes within installations, and a comparison of the Previous Merkle Root as retrieved from ELA/ESC, with the value recorded on the database. The historically true value is readable from Elastos. Any discrepancy causes a warning to be issued to all affected companies/organisations, and affected production is halted while an investigation is conducted. Unless the discrepancy represents a “false positive”, the root cause is determined and appropriate action taken, before production is resumed. All database operators understand this automated surveillance is taking place constantly, so it is unlikely, though not impossible, for malicious interference to occur, but the threat from Hacking persists. Of course these cross-checks insure against hackers.

Additionally, the system enables us to deal with simple machine malfunction which need not be malicious.

The following 2 figures show the plan for deploying and using Elastos Carrier2.

cloudSchematic

A Blockchains such as Elastos represent an immutable source of Truth, alongside an Elastos Distributed Identity Blockchain (DID Sidechain). The latter will mesh with the Postgres ‘DataChain’ (or ‘BlockBase’) to provide indisputable Identification of players across the entire system (so-called Non-Repudiability of transactions). Elastos is a Global Network permissioned Blockchain. Naturally the bulk Corporate Data will belong on the PostGIS databases, but there is a role to be played by the flexibility and mobility of the Elastos system, especially when we consider the strengths of Elastos Essentials and dApps, Carrier’s security, in particular its avoidance of the DNS system, and the DID Sidechain together. So we intend to utilise our own adaptation of the Elastos-Smartweb-Service github repo and docker image to play the role of joining the DataChains systems to the Blockchains systems, by acting as a very smart reverse proxy webserver, database server and blockchain server; all at once. Additionally we have completed development of the financial/critical transaction registration and checking servers, for provision of Fraud Insurance against the databases. The idea is to enforce a “no corps > 33%” policy. Here, a corps represents a body attempting to cause fraudulent transaction(s) to be perpetrated, by directing more than 33% of the Ordering Servers to act fraudulently together. Naturally, more than one person, acting criminally, may be involved in a corps. The registration phase is to be followed by the “checking” phase, where the reported results of Database Block Committals are compared to the records registered on a node-based etcd distributed database at the times of execution (and then committing or rolling back) of transactions. This enforces an honest action of execution and committing/rolling back of transactions, since the etcd records and subsequent Elastos recordings are made both immutably and automatically, in concert with BFT-SMaRt, and the resultant Merkle Roots are cross-checked. Alarms are broadcast to all companies on any inconsistent installation (which is still possible, though policed at 100%, and alarms multicast upon any/every offence detection) followed by the implementation of Contingency Plans. The Database Operators know this, so although remaining vigilant, we expect no breaches from operators, however not so from Hackers. This Ultimate Checking mechanism occurs many times daily, 24/7.

We are also developing an encryption system, which employs customising capabilities included with the Elastos Distributed Identity system, that will enable a per-Contract, Company and Role-within-company-specific set of encryption keys. The fundamental concept here is the use of asymmetric encryption, to allow secure data sharing. As you would expect, no database operators, or members of our own company, are privy to any encryption keys.

cloudSchematic

What companies stand to actually gain from our processes, comes down to the way the Virtual Machines on the database nodes are designed to automatically keep each other honest, with the backup of ensuring Utter Honesty via our proprietary Ultimate Checking System. To explain, our System also operates automatically on those same virtual nodes, policing, at 100%, any “escapes” from the main Trust-Enforcing System (designed originally by IBM and, by our choice, using the University Of Lisbon’s BFT-SMaRt package, but only as effective as any Byzantine Fault Tolerant system can be). With criminal collusion of operators and/or hackers, it would be possible to “override” the IBM/BFT-SMaRt security design. By performing our own checks against trace records, of all original financial and otherwise critical transactions (as defined by you), made on the Elastos Blockchain, we prevent any such “escapes” from being perpetrated without knowledge of the crime(s) amongst all stakeholders. Such crimes will be automatically notified by alarms multicast on the secure Elastos Carrier System, to all those affected, within minutes of being committed. Thus, as long as all Payment Gateways to finance facilities in Suspicious or Rogue States remain blacklisted by default at the Cloud Site, the system is secure, since database operators or hackers will not have access to alter black or whitelists – an activity in the domain of the CIO of customer companies only. We add to this, the impenetrable network transport security provided by Carrier2, which bypasses the traditional Domain Name System (DNS), to emerge with a guarantee of “Total External Security” of all your database and data transport networks, outside your company walls.

As explained on the Agreements page, and at the head of this page, we cannot do more than this as far as providing guarantees, since it always remains possible to defraud a company from within. (But see ‘Elastic Stack’ below).

You may have noticed the reference to “No Operator Read Access” and “No Operator Write Access”. This means no company’s data can be accessed for READ or WRITE purposes by an operator. We were working to employ “Homomorphic” “Client Side Encryption” for all customers. However, apparently, even simple 1-second plain text operations can take more than one week to complete if homomorphic encryption methods are used. So we are returning to a regime where WRITE access by non-users (eg Operators), to user tables, is totally disabled, and there will be no need to notify Company Officers upon the logging of any READ access to a customer’s table(s), since all private data will be encrypted (non-homomorphically) with keys possessed by customer companies alone.

Our systems do require access, however, to some unencrypted datafields in order for database security checking to function properly, but with non-homomorphic client-side encryption for all fields, that do not enter into data security computations. Fortunately, the data access required by our security functions excludes most of your data, as a very tiny subset only is required by our security systems, and none of that seems confidential in any way (at least to us, at this point), being more in the realm of system-technical data. Remember that our GOOD agreement gives your data pride of place (unlike with Microsoft and Google, for example, who are scrambling currently, to be able to analyse your data for their purposes (driven by greed and the amorality of Accountancy), without decrypting it, somehow – in fact anyhow if it can be done legally. This explains the intense research being done into methods allowing a practical equivalent of homomorphic encryption).

For companies in a cooperative network, there will be no need to share encryption keys, since techniques exist allowing separation of encrypted information, although the information may be fundamentally identical (eg contract details). Importantly, Directors, Operators and Employees of ITOTCCA will not have access to any customers’ crypto keys. In the case of companies sharing one member class server (ie an industry-wide, non-internetworked application, such as the Community Housing member-class, or the Real Estate Agency member-class), there will be no need for shared keys in any case. If such an arrangement does not provide adequate security for your needs, we recommend that you consider the more expensive alternative of creating your own singular-company member class(es).

gRPC protocols (using HTTP 2) replace the older style REST APIs, for communicating requests, and data inputs, from the client, and responses from the blockchain &/or database back to the client; all this occurs through the smart-web forwarding proxy server. Here the gRPC protocols are implemented in Typescript (at the clients’ devices) and in Java (in the Cloud). The smart-web server has Carrier2 installed onboard, guaranteeing security. Carrier2 is also installed via plugin on the Ionic dApp clients (which are developed hierarchically inside Elastos.Essentials) as well as manually on the IoT Edge client/servers (running ‘node-red’).

For access to traditional web technologies (eg Banking, Email) we provide the familiar REST API technology on secure channels (HTTPS).

The Internet of Things (IoT) system revolves around the IoT node-red servers on the Edge Client/Servers, being in communication with node-red server residing in the cloud and communicating with the local member-class database. The database receives notifications from the node-red server and passes on relevant IoT events to the blockchains, also communicating via the smart-web service with the IoT Admin dApp.

This system also has the capacity for the databases to emit orders as JSON streams, if the orders (for example) require Operational Technology Domain events to occur automatically (eg manufacturing or production control/initialisation) and to send an order summary (or some other form of production control) as a JSON dictionary-styled object to the Node-Red server, which is then interpretted using node.js. The code is converted to a JavaScript object and operated-on by Node-Red, penetrating into the OT domain safely (possibly involving the Ctrl-X movement automation platform).

The actual “Things” generating messages and signals are connected at the sites by raspberry-pi “edge” clients or similar (eg one of Canonical’s Partners offers “Industrial Pi” with Ubuntu Certification, ready for the Ubuntu “Core” reduced-footprint Operating System). The administration of the IoT system occurs on the IoT dApps, connected by node-red as well as java-grpc to the server(s) and by the IOTA onboard client to the “Tangle” for event validation (as for the node-red cloud server and the edge server). The IoT Admin dApp can be split effectively to allow networks to be under your own control and responsibility.

Both notification and actuation are possible, however with actuation, the response time of the system needs to be considered, as a cloud-based server may not satisfy the sub-millisecond requirements of some machinery. Edge-based servers and mechatronics would be needed to handle such control problems. Systems such as this are communicating Notifications to the Smart-Web Servers for database and blockchain recording purposes (including Data Analysis and AI operations) and to allow IoT Administrators to act as appropriate. As mentioned, the Postgres Database will be able to direct an Enterprise Data Item received which requires IoT (Operational Technology) actuation (eg a Sales Order for manufacturing) to the Node-Red server to be introduced into a production line. This is Flexible Manufacturing.

In summary, the electronic control required is provided either by the IoT devices themselves, or via components like Industrial Pi servers and PLC’s (Programmable Logic Controllers), locally, for groups of devices, but some types of OT process initiation, and IT data collection, can occur through the Enterprise IT system. All communications are shielded securely on networks by Elastos Carrier, requiring the Carrier-Native and its Nodejs and Cordova “wrapper” packages to allow node.js and typescript apps to access Carrier.

ITOTCCA chooses to implement the industrial version “node-red-industrial” of node-red. Both these packages enable the visual programming of connections between IoT devices, edge client/servers and the main IoT server in the cloud, including the specification of node.js functions and system calls to be generated upon events, all varietiies of networked intercommunications, ordering of connections, etc. The main differences between node-red and node-red-industrial are concerned with Industry 4.0 readiness, with the industrial version implementing such features as an interface for Bosch-Rexroth’s ctrl-X automation-of-movement Operational Technology system, for factories of the future.

Usually IoT notifications are dealt with on a “Report-by-Exception” basis, at the cloud level, however a continuous set of signals is often required at the edge level, to maintain control. For AI-driven data analysis much more data can be collected.

The technology of the fee-free IOTA “Trust Layer”, being developed for IoT, is already at version 2.0 and has plans to implement Smart Contracts (presumably fee-less also). ITOTCCA is incorporating IOTA as our provided Trust Layer of choice for IoT operations. IOTA adds a layer of reference against which all IoT actions can be validated. The absence of fees is made possible by having each transaction validate 2 others before completion. In this way there is no “Mining” involved, as such. IOTA is designed to support Industry 4.0 Standards, and is currently popular in Europe (due to the German-originated Industrie 4.0 standards, with IOTA’s headquarters in Berlin. Language translation is natively provided in German, French and Italian, as well as English). The IOTA response cycle-time is measured in seconds.

Simplified examples of production flows involving Node-Red-Industrial are shown below. Here the Head company initiates some Jobs, and we show a range of possible control and communication flows within Supplier companies in the second and third shots. Job 1 is a product test, while Job 5 entails the manufacture of 3 sub-assemblies, by 2 different supplier companies. Job 2 comprises a somewhat more complex set of Electro-Mechanical Unit Tests.

We utilise one “oseer” schema per member-class, so there is one schema for each of the n x m member-classes across all sub-networks, and one for each of the F(i) non-internetworked future dApps, as well as one for each of the A, B and C dApps (Community Housing, RealEstate and convey-IT, respectively). The “oseer” schemata provide Business Process Control for each Member-Class dApp. Our schema called ‘das_fuhrwerk’ is the Head Schema, for top level customer on-boarding, usage tracking and payments, and to provide our own company’s Enterprise dApp for our accounting, planning and administration purposes.

Although “top-level” to us, in this automatic Trust scenario, ‘das_fuhrwerk’ is equal with every other member-class on every sub-network, as far as appropriate privileges are concerned, and is operated under the business processes supervision of its own “oseer” schema. The scope of control of the dApp connected to the ‘das_furhwerk’ system is identical to every other member-class, with the exception of its ability to get reports on user (employee) numbers in each month, on some sub-networks, for billing purposes.

The real basis of separation of Business Channels, is achieved via the Permissions systems including RBAC in Kubernetes, the Postgres User/Role system and the Ionic permissions system. These systems allow fine grained control of access to menus, data, etc. As the system grows, a single Business Network running separately-permissioned Business Sub-Nets, would be separated, eventually, into multiple Business Networks (with id == i), containing sub-nets (id == n), with member classes (id == m). A node would therefore have a full id of “Node(i,n,m)” with individual (“Member” or Company) business channels denoted as j on that node. A single Company’s full id would therefore be “Member(i,n,m,j) (of Member-Class (i,n,m))”. Take note that one business sub-net (“n”) is equivalent to one business system, say a single supply-chain (for example a supermarket supply chain or a petrochemical supply chain).A single business system may cross 2 or more “installations”, although there can also be installations containing more than one business system.

In an Ionic dApp, such as we use for the Front End GUI of any Elastos dApp, (either React or Angular-based), you will find manifest.json.

This file is where you can whitelist websites (including the cloud database/blockchains and server site). All websites are blacklisted by default, until you whitelist the sites required by your dApp. This enables the Elastos Carrier system to do its magic and connect in a private and ultra-secure way to websites, with no websockets exposed.
(see the Elastos Whitepaper at https://www.elastos.org/downloads/elastos_whitepaper_en.pdf).

Something of the potential of this project is seen here, in the Docker version we are also developing, mostly because we can scale this on a desktop computer far more than using Kubernetes.

Replicação da máquina de estado bizantina para as massas

Byzantine State Machine Replication for the Masses

We have incorporated a total of 29 checks (0 through 28) in ‘ChubbyChecker’.
At the completion of Registration of the Kth Block (ie at the end of each Ultimate Registration Phase), we begin the following set of 29 Checks:
Check ALL the following conditions are MET, else ALARM STATE:
(Note that currently the number of Blocks
in a single Ultimate Registration/Checking Phase, K = 24,
and there are designed to be 1024 Transactions per block)

Check 1: Node Identities: Verify that the nodes involved in the consensus process are the expected, authorized nodes.
Check_2. Across all transactions, in Post-Registration-execute and in Post-Registration-commit/abort phases, block sequence numbers must be continuous.
Check_3. In the Pre-Registration files, ostensibly conitnuous among the K Blocks, the Traces cannot be divided into Blocks a priori. However by writing a block marker file to the etcd/DHT system when each block (ie decided by machinic consensus) is denoted and recording the TxUUIDs contained in the block, and writing an encrypted file with that Smart Contract’s Id, locally (ie to the vm or “installation”), ChubbyChecker can assign traces, from the Preregistration Stage, to blocks, and check Block continuity a posteriori. Using this method, on each node, the total of Preregistered (submitted) Transactions in one Ultimate Checking Phase must equal (total of TxUUID’s executed) – in the Post-Registration-execute phase – AND must equal (total of TxUUID’s committed + total of TxUUID’s aborted) in the Post-Registration-commit/abort phase, across all Post-Registered Blocks in the same Checking Phase.
Check_4. Across all nodes, and for both Post-Registration-execute and Post-Registration-commit/abort phases, for a set of transactions in BlockSeq (k), with TxUUID = z: ([Total of TxUUID’s with opCodeExec == “true”] == [Total of TxUUID’s with opCodeCommit == “true”] OR [Total of TxUUID’s with opCodeExec == “true”] == [Total of TxUUID’s with opCodeAbort == “true”]) AND len{TxUUID = z} = N(i), the Number of nodes in the installation [i] being examined , for all (k, z) in [i]. (Transactions in k determined a posteriori from phase (c), as in Check_3).
Check 5: Block Hash Integrity: Validate that the block hash matches the expected value, confirming that the block has not been tampered with.
Check 6: Transaction Hash Integrity: Verify the integrity of individual transactions within the block by checking their hashes.
Check 7: State Consistency: Ensure that the state after each block execution is consistent across all nodes, reflecting correct and deterministic execution.
Check 8: Commit Consistency: Confirm that the blocks committed by different nodes are identical, with no divergence in the blockchain.
Check 9: Byzantine Fault Detection: Analyze any discrepancies in node responses to detect possible Byzantine behavior.
Check 10: Message Authenticity: Verify that all messages exchanged between nodes during the consensus process are authentic and correctly signed.
Check 11: Network Partition Detection: Detect any network partitions or communication issues that might have influenced the consensus process.
Check 12: Latency and Timing Consistency: Ensure that the timing of block commits and transaction execution is within expected bounds, identifying any anomalies that could indicate manipulation or delays.
Check 13: Redundant Storage Consistency: Verify that redundant copies of transaction data and state information are consistent across nodes.
Check 14: Recovery Point Integrity: Ensure that the recovery points or checkpoints created during the process are accurate and can be used to restore the system state if needed.
Check_15. Across all checked transactions (Pre-Registration-submit, Post-Registration-execute and Post-Registration-commit/abort phases), each signature is traceable to a current client from the current installation’s range of member-class servers’ (ie nodes’) clients.
Check_16. Across all checked transactions, the ‘entnum’ field in the transaction’s contents, when compared to the client’s registered ‘entnum’, must match.
Check_17.Across all checked transactions, the ‘schema’ field in the transaction’s contents, when compared to the client’s available ‘schemata’, must match one item. These are fully qualified names ie database.schema.table.
Check_18. The “amount” field in all financial transactions, and where amounts exist, in any other transaction, are constant across preregistration (submission), execution and committal/roll-back.
Check_19. On each node, TxUUIDs are generated at Pre-Registration, so in each of the k BlockSeq numbers, for (a). Pre-Registration-execute phase, (b). Post-Registration-execute phase and (c). Post-Registration-commit/abort phase, each TxUUID occurs once exactly, and the TxUUIDs in (a) are “1 to 1” AND “onto” the sets for (b) and (c). See Check_3 above for method of distinguishing blocks.
Check_20.On each node, across phases (b) and (c), [Total of TxUUID’s with opCodeExec == “true” (which should be all TxUUID’s)] == [Total TxUUID’s with opCodeCommit == “true”] + [Total TxUUID’s with opCodeAbort == “true”].
Check_21. On each node, the total number of structs in the array of structs, spanning each BlockSeq (k), is equal to the agreed number (eg 1024) of transactions per Block, and this holds for all of (a). Pre-Registration-execute, (b). Post-Registration-execute, and (c). Post-Registration-commit/abort phases. Here the number of transactions in each BlockSeq for phases (a) and (b) is determined a posteriori by reference to phase (c), as in Check_3.
Check_22. Across all nodes, at the Post-Registration-commit/abort phase, for a full set of transactions in BlockSeq (k): the ordering of transactions must be constant for all member class servers (nodes), and for all k under examination.
Check_23.The value of the previous Transaction Trace Merkle Root is retrieved from the Smart Contract and compared to the value recorded on PostGIS. The values must be identical.
Check_24.We then rebuild the Merkle Tree for the previous slab using the transaction trace database’s history and compare the rebuilt root with the one retrieved from the blockchain. If the rebuilt root differs from the blockchain root, it could indicate tampering in the underlying transaction data. This approach verifies not only the stored root but also the integrity of the transactions that contributed to it.
Check 25:We validate that every transaction contributing to the current Merkle Tree has a corresponding entry in the metadata store. We ensure that these entries match the expected data (e.g. transaction IDs, hash values).
Check_26.Instead of relying solely on the Merkle Root, we include “hashes” of individual transactions in the leaves of the Merkle Tree. This ensures that even if the Merkle Root is consistent, individual tampered transactions can be identified by checking their hashes.
Check 27:We use periodic snapshots of the database, stored securely in an independent location, to cross-validate the database state. Snapshots are used to detect discrepancies that might otherwise go unnoticed due to tampering with live data. Thus we take database snapshots at the end of each slab. These snapshots are securely stored using immutable storage. We use the snapshots to rebuild and validate historical Merkle Trees
Check_28.Proof-of-Consistency algorithms (like those used in blockchain systems) are used to ensure that:
- The current Merkle Root is consistent with the previously recorded root.
- No invalid transactions have been added between slabs.

Check_0

For full Security Compliance, this Check, along with Checks _1 to _28, can only be performed after all proposed database alterations have been completed.

As soon as any of these checks returns a positive Fault, an Alarm is multicast, whereupon the databases in the offended installation are shutdown gracefully, IoT operations for the relevant installation(s) are overridden and set to “Operation without Database”, and then the affected databases are restarted in ‘Investigation’ mode, with ordinary users and operators excluded.

You can consult the Forensics page, for a detailed analysis of our alignment with modern Transaction Trace Recording and Cryptography practices.

All the preceding checks have been implemented as GO! functions within the ChubbyChecker package, without full cooperation of the database operations, until after completion of source code alterations.

The “Das_Fuhrwerk” server(s) in the offended installation(s) is the basis from which we conduct investigations (it has an ‘investigations’ table). It falls to Executive Directors and the CEO, CFO and CIO to conduct the investigations. Database, ChubbyChecker and ChubbyRegister log files are examined. The alarm will have been issued with a Code attached. The meaning of the code accompanies the announcement, and will correspond to one or more of the checks outlined above. It is not possible, obviously, to predict the course and duration of the Root Cause Analysis.

The idea is to utilise the initial buffered command line input prompts, in main.go, for trusted staff to issue Status Update broadcasts as the investigation proceeds. All data entered into the Investigations Table is reviewed by ITOTCCA directors and is made available to Officers of Companies involved in the offended Installation(s), to check transparently the logic of conclusions drawn and evidence provided. Note that the first line of examination attempts to define whether the fault constitutes a “False Positive”.

Normally, only 1 installation would be in an “offended” state at any time, however we do not rely on this. Multicasts are restricted to officers of companies within offending/offended installations.

URL for IBM article

Please note there are 2 possible senses of the words Order and Orderer. One is intended to refer to the ordering of Health supplies. This is not the sense intended in the following paragraphs. The sense intended is the Ordering of Transactions – ie putting submitted transactions into an order determined by a consensus process on a virtual machine, and then committing or aborting them in that order. The order will appertain to each Virtual Machine independently.

An example of a Fully Developed National Database Network

The tableau shows an array of 22 Hardware On-Prem Canonical OpenStack Cloud Servers, positioned around the nation at State and Territory locations (minimum 8 in a fully developed network) and at Royal Flying Doctor Entity locations (minimum 7), in addition to a range of 7 Hardware Servers representing the 7 Supplier Classes.

These cloud servers are interconnected and designed to replicate all transactions faithfully across all
7×7 + 8×8 + 7×7 nodes (162 nodes).

The extended diagram below gives an indication of the nature of replicating from each client’s requests, across all networked nodes, using a single client as an example. Each client submits transactions to their own V.M./load balancer which is multicasting each transaction to all other V.M’s in the entire national network (ie to the other 22 – 1 = 21 load balancers). Each load balancer is programmed to be aware of its own expected local client id’s from the range of clients “belonging to” that load balancer. Transactions from these clients are multicast across the network, as well as participating in their local round robin process. When a load balancer receives a transaction request from an “external” client, (ie not on the list of local clients “belonging to” that load balancer/V.M.), it distributes this request to an inernal node, (determined by the current position in the round robin process for that load balancer), on its V.M., without further multicasting the request.

Whilst the various load balancers are engaged in a round robin process of distributing transactions upon receipt to one single node in its own V.M., and, for “local” requests only, multicasting to the remaining V.M.’s across the national network, it is the BFT-SMaRt ordering and replicating packages (ie ‘the orderer nodes’) which multiply (replicate) the transactions received on each node, across all nodes in a V.M., each holding the status of a Master Copy, and order the full set, defined in terms of Blocks, before executing then committing/rolling-back the transactions in the Block. The size of the Blocks (currently 1024 transactions) is determined by software written by ITOTCCA, as is the number of Block Commitments (currently 24) in an Ultimate Checking phase involving ChubbyChecker.

Our Virtual Machines

A different perspective is provided by the Social Housing, Real Estate and Construction Industries

One obvious question is “Will the variety of 22 (or 42 in the Construction Supply case) Virtual Machines all replicate their received transactions in the same order?” The answer is that, due to the independence of each V.M., and the asynchronicity of the system, their orderings will be different. However, we look after these discrepancies after every 24th Slab is checked by examining the 24 processed Slabs (multiplied by the number of each Enterprise’s Network-wide clusters) to extract the transactions which have been processed differently between clusters. There are several possible important differences:
A transaction may have been rolled back in one or more clusters without evidence of reprocessing and committing successfully in some of those clusters. As long as there is at least one other cluster which committed this transaction successfully, the transaction is forcefully committed in all exceptional clusters

A transaction may be omitted in one or several clusters (ie not executed) despite being executed AND committed in at least one other cluster. These transactions are force-replayed (ie executed, if necessary repeatedly, until successfully committed). The events where all the other clusters involved have executed but not committed a transaction, are not acted upon

Variations between clusters in the order of transaction execution, when followed by commitment, can not give rise to anomalies. ITOTCCA follows IBM’s use of the concept of SSI (Serialisable Snapshot Isolation) and the associated methods, to assure database consistency, which we employ to ensure consistency during Block commitment procedures. This involves the related notions of keeping lists of “In Conflict” and “Out Conflict” transactions, and of “near” and “far” conflicts, and the associated rules using these definitions. You may refer to this page for expositions.
Finally, we observe that if every virtual machine is kept internally and externally consistent, then the entire network is also internally and externally consistent, despite variations in the order of committed transactions, and, with the latter synchronised 24-Slab examination, even including the possibility that different transactions may be rolled-back on the databases of different virtual machines. The independent internal and external consistency of the various machines’ databases is still a guarantee against non-insider fraud, noting that ITOTCCA never would be able to offer guarantees against “insider jobs”, even for only one stand-alone virtual machine, if we class “insiders” as any person granted permission to register with the Elastos DID system, and authorised to submit any particular class of transactions, and therefore able to sign those transactions. The guarantee is qualified by the fact that we process each Critical Transaction individually (where our clients define “Critical” Tables), but we only perform Batch Processing of transactions classified as “Non-Critical”, in order to balance performance and cost against residual risk.

ChubbyChecker thus ensures the integrity of the states of all databases in the network at the end of each 24 Block Commitments, as determined independently by each virtual machine (meaning there would be up to 22 different transaction-orderings and 22 different points at which 24 Block commitments will have elapsed).

By performing a full set of 29 internal consistency cross-checks to verify consistency across nodes in a virtual machine, and also,
By performing, at the same stage, a comparison of the prior database-recorded Merkle Root, with the latest Trace of Traces (Merkle Root) on the Elastos Blockchain, thus constituting a verifiable audit trail, and providing a means of checking for external consistency of the database state, at this Ultimate Checking stage just prior to recording the current Merkle Root.

Thus there are 22 (= number of V.M.’s in the network) possible independent origins of Alarms indicating faults, in this representation of a fully developed National Network. Also, the network is extensible, due to the independence of the V.M’s, and need not encompass 22 Hardware servers from the beginning. It is possible to view the independence of V.M.’s, therefore, as an advantage of the design. BFT-SMaRt allows for this extensibility (under the heading “Reconfiguration”) in its own design.

One of the significant advantages of these processes, including the use of our ChubbyChecker with the Byzantine Fault Tolerant process provided by BFT-SMaRt, together with the use of client-side encryption, Role Based Access Control (RBAC), and the offer of asymmetric cryptography where legitimate and agreed, is that business partners, in this case, Governments in partnership with their suppliers, do not need to trust each other in order to gain the benefits of participation in automated and secure trading including full accounting. For the same matter, neither do Federal, State and Territory Governments need to trust in each other, nor do suppliers need to trust other suppliers.

On the subject of accounting, as far as accounts pertaining to Health Supply purchasing and related items are affected, we must insist that Government Health Departments be prepared to cross-over to our proprietary Accounting System, the multiplexed ‘Block ‘n’ Tackle’, which allows for multi-party transactions. We cannnot allow other than machinic read-only interfaces to our system, for security reasons. This means both the consumers (Health Departments and RFDS Entities) and the business suppliers in the system, are required to participate in accounting systems provided by ITOTCCA’s Block ‘n’ Tackle, for transactions concerning this Health Supply Network. The interfacing of our system to Governments’ and Business’ broader systems must be on a machine-read-only basis. At this point it is worth remembering that Governments and businesses hold their own encryption keys to their own data. We also refer you to our “G.O.O.D.” Agreement

Our Block ‘n’ Tackle accounting system will be open for thorough testing before going ino production. We have completed the skeleton structure and functioning of the system, in the form of “update_master()” and “update_tax()” functions in PlpgSQL (the Postgres procedural language) and tested and proven these using Java client-functions. The details of the fields for the Journals and Ledgers must be completed during contracted development.

The networking of databases also enables the sharing of information aside from the operation of Smart Contracts for trading in goods. The way is open, with the replication of information across databases and the availability of asymmetric cryptography, to temporarily grant permission, say, for an entity of the RFDS to access patient information, legally, by simply encrypting the necessary data with the public key of the receiving entity, and sharing the data with the RFDS (via the secure Carrier2 Peer to Peer Elastos Business Network – avoiding insecure email transfers). The convenience and security of the total replication system, provides a way for the RFDS, in this case, to access external data, legally, on their own servers.

Obviously this also means that information may be shared, when legally permissible, via the same means of asymmetric cryptography, between State Health Departments and RFDS entities alike. A similar process would occur for the shared data involved in Smart Contracts for the supply of Goods and Services. The advantage of using databases instead of blockchains like Ethereum, lies partly in the volumes of data that can be handled on databases, as well as lying in the non-public nature of such database networks, in comparison to blockchains.

As stated elsewhere on this page, we are prepared to donate all these software-based services to the Royal Flying Doctor Service, as a full participant.

So, with regard to the integration of Carrier 2 with BFT-SMaRt, we first need an extant SuperNode Network (Bootstrapping Nodes), and a committed process for any node requesting to be bootstrapped onto an authorised Network, in response procedures on the SuperNodes (such as those involved in the BootstrapCommand class). The request process on the Database Nodes must exist also.

As part of a sustained effort to eliminate reliance on DNS, IP Addressing and the use of standard TCP Ports, we have commenced redevelopment with QUIC (Quick UDP-based Internet Connections). QUIC uses TLS 1.3 on HTTP3. Together with gRPC, the combination yields fast, low-latency connections with increased security and is compatible with our Kademlia DHT network and (with the help of etcd distributed storage) does allow for Peer Discovery and Connection without DNS. Pure embedded DHT. We are currently removing all dependencies relating to DNS, IP Addressing and TCP Ports across all our components.

And for Aerospace Industries:

Executive Summary

Summarising, what you get is a system that enables automated trading using Smart Contracts that are enacted on a private network, and where, due to the ability to share information with supply/health partners, using asymmetric cryptography, and also due to all database clusters acting as Masters in replication procedures, a partner organisation can access the shared data on their own local server. Sharing may also be completely revoked.

You also get a comprehensive Enterprise Application that works on Mobile devices and provides all accounting as part of the package, where the database we use (PostgreSQL) has zero licence fees, yet we can offer a 99.999% uptime guarantee (with Enterprise DB Support), excluding any down time due to the perpetration of “external” fraud (which will be detected by ChubbyChecker within minutes) and the subsequent investigations. The accounts system can be interfaced in a READ-ONLY fashion to your broader system. Or you may choose to convert all accounting to our system – which is quite possible and will be cheaper, and more secure, in all likelihood, than your present systems.

In addition you receive an economical and very security-conscious system, incorporating our ChubbyChecker Ultimate Cross-Checking package, which fills the security gap inherent in stand-alone distributed systems that employ Byzantine Fault Tolerance, as well as all the normal security measures such as client-side end-to-end encryption, together with the absence of risk of Man-In-The-Middle and Denial-Of-Service attacks (with our use of jvm-libp2p, bypassing, as it does, the normal Domain Name System – DNS). The distributed nature of the system is what allows the operation of private-networked Smart Contracts, incorporating unlimited volumes of data per contract (and overall), and also it is what allows the secure sharing of information, as necessary, to enable business/health operations to function efficiently.

But what is the magnitude of the risk against which we are protecting? According to an independent analyst organisation (Analysis) there were 25,949,000 personal records compromised in the healthcare sector in 2023 alone. “These are often called ’supply chain attacks.’ In fact, it is not just one customer that can be attacked, but essentially every customer that uses that vendor’s services. Thus, a single vulnerability can threaten many thousands of organizations, such as in the 2023 MOVEit attack where already over 2,600 companies in more than 30 countries admitted to being attacked, with more than 80 million individuals’ data being compromised. In fact, 98% of organizations have a relationship with a vendor that experienced a data breach within the last two years. In some studies, the number of data compromises due to supply chain attacks in 2023 jumped 78% over 2022.” Stuart Madnick – Harvard Business Review The problem is no longer a case of “if” but now a case of “when” will your systems be attacked.

There are many vectors by which attacks may be mounted. Our use of jvm-libp2p avoids some, however, still, the most common and “fruitful” means of attack involve targeting individuals using Social Engineering. As an example of Social Engineering, the attackers may use knowledge gained in any fashion about an employee’s vulnerabilities, contacting that employee and using this knowledge to obtain the employee’s cooperation in an attack or in the exfiltration of information from your database. It is very important to have multiple offline data backups in place in the case of an attack, which may also arrive in the form of Ransomware. The basic process involves the attackers accessing your data and re-encrypting the data to prevent your access, then demanding a Ransom for decrypting the data. If you have a good back-up system, and your data is strongly encrypted, this is simply not a problem. Note that the backups need to be kept offline and thus inaccessible to an infiltrator online. Also consider the number of people who send passwords via email to themselves as a record, or to others to access their account in an “emergency”. If these emails are compromised in external third-party attacks outside your system, your system may become compromised by a malicious agent. This touches on the concept of an Attack Surface, meaning the myriad vulnerabilities any system may have.

Thus, there is a multitude of possible attack vectors threatening your system. Sealing it with the use of jvm-libp2p goes a long way towards total security. But there are many other practices that should be used to defend against Hacking and System Penetration. We can only offer advice to be vigilant and aware of the size of the threat, and to take appropriate counter measures (and there are many measures to take). As noted elsewhere our use of client side encryption will assist, as will the frequent, programmed running of our ChubbyChecker. Insider jobs via Social Engineering techniques and third-party breaches are still high on the list of threats. Education of all staff can help here, as can monitoring of database access patterns, and vigilance (automated) against abnormal patterns of use. Also, the use of Artificial Intelligence in this monitoring can make a difference when it comes to detection of “insider jobs”. An AI system can be trained to recognise excursions from normal usage patterns, and its performance is actually enhanced over time as it gains “experience”.

ADNS – Absence of DNS – Enterprise System

ChubbyChecker: Elastos ELA/ESC Blockchain
Blockchain vs Database:

Factory 0:

Factory 1:

Factory 2:

Elastos Blockchain

Bucordo Security (BFT-SMaRt)
merged with jvm-libp2p and:

“ChubbyChecker“

Elastos Carrier2 Transport, End-to-End Encryption,
7-Stage x 29-Factor Cross-Checking,
No Operator Read Access,
No Operator Write Access

The Checker Anchors
Merkle Roots to Blockchain
and stores all Transaction Trace
evidence on WORM Drive

BFT-SMaRt jvm-libp2p (Kademlia Protocol)

QUIC Transport gRPC API to/from the Edge

in the DataCentre Global DHT (Peer Discovery by Id)

Main Enterprise Data on PostGIS
Merkle Roots Anchored All Transaction Trace Evidence Stored
Immutably to Blockchain .. to Write-Once Read-Many Drive

________________________________________________________________________

‘BFT/DHT Bucorde’™

Twinned Networks

An Absence of DNS (ADNS) Enterprise System

Bucordo/BFT-SMaRtDatabase and Server Nodes

[Supernodes, Secondary Nodes]

Kademlia/jvm-libp2p/DHTDistributed Hash Table Nodes

[Supernodes, Secondary Nodes,Tertiary (Client) Nodes]

VS

With ChubbyCheckerAnd SuperCheckerUltimate TransactionCross-Checking

“ChubbyChecker“.

QUIC/gRPCHTTP3TransportLayerPeer Discoveryby NodeId

TLS 1.3TransportEncryptionTotal Absence of DNS~ ADNS ~

________________________________________________________________________

Merging adapted jvm-libp2p SuperNodes with Bucordo PostGIS Database NodesAnchored to Blockchain© by IT/OT Chain & Cloud Australia Pty Ltd 2022-2025

You might call it ‘100% FAULT INTOLERANCE for Databases’(with In-Transit Security)

________________________________________________________________________

ALL DATA IS ENCRYPTED BOTH IN TRANSIT AND AT REST

GENERALLY USING YOUR OWN PRIVATE KEYS, BUT USING OUR PUBLIC KEY,

OR PARTNERS’ PUBLIC KEYS, TO SEND US CHANNELED, PRIVATE COMMUNICATIONS

High-Level Architecture Diagram: BFT-DHT with jvm-libp2p Backbone

~~~~~~~~~~~~~~~~~~~~~~~

Roadmap Checklist: Immediate & Near-Term Milestones

Now (Completed or Near Completion)

Bucordo/BFT-SMaRt
Database and Server Nodes

Kademlia/jvm-libp2p/DHT
Distributed Hash Table Nodes

[Supernodes, Secondary Nodes,
Tertiary (Client) Nodes]

With ChubbyChecker
And SuperChecker
Ultimate Transaction
Cross-Checking

QUIC/gRPC
HTTP3
Transport
Layer
Peer Discovery
by NodeId

TLS 1.3
Transport
Encryption
Total Absence
of DNS
~ ADNS ~

Merging adapted jvm-libp2p SuperNodes
with Bucordo PostGIS Database Nodes
Anchored to Blockchain

© by IT/OT Chain & Cloud Australia Pty Ltd 2022-2025

You might call it ‘100% FAULT INTOLERANCE for Databases’
(with In-Transit Security)

High-Level Architecture Diagram: BFT-DHT
with jvm-libp2p Backbone

Roadmap Checklist:
Immediate & Near-Term Milestones