Creating 400,000 nfts takes 30+ hours

[anvabr]

Description

Our use-case (environmental - carbon units/offsets etc) requires the creation of large number of NFTs which represent these carbon units. Realistically we are unable to achieve objective as it takes more than 30 hours to create 400,000 nfts using the SDK facilities. We attempted to parallelise the execution however it resulted in duplicate transactions.

Steps to reproduce

Use TokenMintTransaction and TransferTransaction SDK facilities to mint a large number of NFTs and transfer them to the owner[s]. Since these API methods can work with up to 10 records, in our use-case it makes 40,000 transactions.

Additional context

No response

Hedera network

testnet

Version

N/A

Operating system

No response

[rbair23]

Can you post a reproducible example? Currently the network is configured with a setting of 1200 mints per second. The throttles are applied per-node as 1200/N where N is the number of nodes in the network. So if you are testing with a 7 node network and only sending traffic to a single node, then we would expect to see a throttle of 1200/7 or about 171 mints per second, which should still take less than an hour to complete. If you are testing on a 26 node network, then that would be about 46 mints per second or several hours to complete.

[anvabr]

@rbair23 thank you for the reply. Just to clarify, when you refer to a networks with a 'number of nodes in the network" do you mean a private test network? We are using public testnet, which I think is a 7-node network.

Do you have scalability/performance tests or sample code that we can take a look and ideally run which demonstrates the performance you are describing? Could you direct us to it please?

[gregscullard]

To efficiently parallelise the operation, you should ideally use a different account Id per "thread" in order to avoid the duplicate transaction ids.
Alternatively, each thread can set its own transactionId per transaction, overriding the SDK such that the nanoseconds portion of the transactionId's valid start is in a different range per thread
e.g.
thread1, nanoseconds between 0 and 99999 (increase from 0 by 1 with each transaction)
thread2, nanoseconds between 100000 and 109999
and so on (alter the range of nano seconds depending on the number of threads)

You may also separate submitting transactions from getting receipts into separate threads, this will ensure that each mint doesn't wait on the receipt to issue the next one.

[anvabr]

@rbair23 and @gregscullard Thank you for your comments. We can't use separate account ID per thread as it is not compatible with our use-case. And when we tried to parallelise with the single account we did indeed get the duplicate transaction IDs.

Here is the extracted code sample which demonstrates the problems. Can you please point to changes that would need to be made here to achieve the rates your are listing above.

const client = Client.forTestnet();
    client.setOperator(OPERATOR_ID, OPERATOR_KEY);
    const operatorKey = PrivateKey.fromString(OPERATOR_KEY);
    const privateKey = PrivateKey.generate();
    const publicKey = privateKey.publicKey;

    const transaction = new TokenCreateTransaction()
        .setTokenName("Your Token Name")
        .setTokenSymbol("F")
        .setTokenType(TokenType.NonFungibleUnique)
        .setTreasuryAccountId(OPERATOR_ID)
        .setInitialSupply(0)
        .setAdminKey(privateKey)
        .setSupplyKey(privateKey)
        .setMaxTransactionFee(new Hbar(30))
        .freezeWith(client);
    const signTx = await (await transaction.sign(privateKey)).sign(operatorKey);
    const txResponse = await signTx.execute(client);
    const receipt = await txResponse.getReceipt(client);
    const tokenId = receipt.tokenId;
    console.log("The new token ID is " + tokenId);

    const data = [new Uint8Array(Buffer.from("f44dce5b-3b4d-4ea6-aae9-2549bd30f9a2"))];
    const max = 1000;

    //1
    console.time('Start 1');
    for (let i = 0; i < max; i++) {
        const transaction1 = new TokenMintTransaction()
            .setTokenId(tokenId)
            .setMetadata(data)
            .freezeWith(client);
        const signTx1 = await transaction1.sign(privateKey);
        console.log('Execute:', i);
        const txResponse1 = await signTx1.execute(client);
        const receipt1 = await txResponse1.getReceipt(client);
        console.log('Executed:', i);
    }
    console.timeEnd('Start 1');

    //2
    console.time('Start 2');
    let count = 0;
    for (let i = 0; i < max; i++) {
        const clientN = Client.forTestnet();
        clientN.setOperator(OPERATOR_ID, OPERATOR_KEY);
        const transaction1 = new TokenMintTransaction()
            .setTokenId(tokenId)
            .setMetadata(data)
            .freezeWith(clientN);
        const signTx1 = await transaction1.sign(privateKey);
        console.log('Execute:', i);
        signTx1.execute(clientN).then((txResponse1)=>{
            console.log('Executed:', i);
            txResponse1.getReceipt(clientN).then((receipt1)=>{
                count++;
                if(count==max) {
                    console.timeEnd('Start 2');
                }
            }, (error)=>{
                count++
                console.log(error);
                if(count==max) {
                    console.timeEnd('Start 2');
                }
            })
        }, (error)=>{
            count++
            console.log(error);
            if(count==max) {
                console.timeEnd('Start 2');
            }
        });
    }

    await wait(600000);

Case 1 runs at about 1s per transaction. In case 2 we get two types of errors if there are more than 200-300 transactions.

StatusError: transaction 0.0.1548173@1647361784.722423145 failed precheck with status INVALID_NODE_ACCOUNT
    at new PrecheckStatusError (d:\work\vctest\node_modules\@hashgraph\sdk\lib\PrecheckStatusError.cjs:23:5)
    at TokenMintTransaction._mapStatusError (d:\work\vctest\node_modules\@hashgraph\sdk\lib\transaction\Transaction.cjs:1068:12)
    at TokenMintTransaction.execute (d:\work\vctest\node_modules\@hashgraph\sdk\lib\Executable.cjs:420:22)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {name: 'StatusError', status: Status, transactionId: TransactionId, stack: 'StatusError: transaction 0.0.1548173@16473617…ions (node:internal/process/task_queues:96:5)', message: 'transaction 0.0.1548173@1647361784.722423145…ed precheck with status INVALID_NODE_ACCOUNT'}

Or

StatusError: transaction 0.0.1548173@1647361778.618899118 failed precheck with status DUPLICATE_TRANSACTION
    at new PrecheckStatusError (d:\work\vctest\node_modules\@hashgraph\sdk\lib\PrecheckStatusError.cjs:23:5)
    at TokenMintTransaction._mapStatusError (d:\work\vctest\node_modules\@hashgraph\sdk\lib\transaction\Transaction.cjs:1068:12)
    at TokenMintTransaction.execute (d:\work\vctest\node_modules\@hashgraph\sdk\lib\Executable.cjs:420:22)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {name: 'StatusError', status: Status, transactionId: TransactionId, stack: 'StatusError: transaction 0.0.1548173@16473617…ions (node:internal/process/task_queues:96:5)', message: 'transaction 0.0.1548173@1647361778.618899118…d precheck with status DUPLICATE_TRANSACTION'}

[gregscullard]

@anvabr something along those lines should fix the DUPLICATE_TRANSACTION

    const thread = 1;
    const nanosOffset = thread * 1000000;
    for (let mintIndex = nanosOffset; mintIndex <= nanosOffset + 100; mintIndex++) {

        const seconds = Math.round(Date.now() / 1000);
        const validStart = new Timestamp(seconds, nanosOffset);
        const transactionId = TransactionId.withValidStart(AccountId.fromString(process.env.OPERATOR_ID), validStart);

        const response = await new TokenMintTransaction()
            .setTransactionId(transactionId)
            ...        
    }

each thread (const thread = 1) uses a different range of nanoseconds, adding one to the last with each mint transaction.

Must be careful not to exceed the max value of a javascript number

[anvabr]

We will test it and come back to you @gregscullard , thank you for the suggestion!

[anvabr]

@gregscullard Thank you for the suggestion on fixing the DUPLICATE_TRANSACTION issue, we've used your design for tests and it appears to have resolved the immediate issue. However we now get Hedera error, which looks like as if we are DOSing it with the volume of requests (see below). Any suggestions here please?

GrpcServiceError: gRPC service failed with status: UNAVAILABLE
    at e.rpcImpl (C:\Users\tmf\Projects\WaveAccess\guardian\threads-test\node_modules\@hashgraph\sdk\lib\channel\NodeChannel.cjs:81:18)
    at e.rpcCall (C:\Users\tmf\Projects\WaveAccess\guardian\threads-test\node_modules\protobufjs\src\rpc\service.js:94:21)
    at executor (C:\Users\tmf\Projects\WaveAccess\guardian\threads-test\node_modules\@protobufjs\aspromise\index.js:44:16)
    at new Promise (<anonymous>)
    at Object.asPromise (C:\Users\tmf\Projects\WaveAccess\guardian\threads-test\node_modules\@protobufjs\aspromise\index.js:28:12)
    at e.rpcCall (C:\Users\tmf\Projects\WaveAccess\guardian\threads-test\node_modules\protobufjs\src\rpc\service.js:86:21)
    at e.mintToken (C:\Users\tmf\Projects\WaveAccess\guardian\threads-test\node_modules\@hashgraph\proto\lib\proto.js:1:276552)
    at TokenMintTransaction._execute (C:\Users\tmf\Projects\WaveAccess\guardian\threads-test\node_modules\@hashgraph\sdk\lib\token\TokenMintTransaction.cjs:209:26)

[janaakhterov]

Converting this issue into a discussion since it's not an issue with the JS SDK.

[anvabr]

@danielakhterov Could you, or anyone else, please address our mains issue? As per the title of the tickets, the question is: should it be the case (or not) that creating 400,000 NFTs takes 30+ hours? Perhaps @gregscullard would be able to comment again?

If the answer is 'no, it should take significantly less' then the second question would be as follows: do you have or know about the scalability/performance tests or sample code that we can take a look at and/or ideally run which demonstrates the faster performance please?

[janaakhterov]

Creating that many NFT's does sound like it'll take a significant amount of time to me, so I think everything is functioning as expected. However, I may also be wrong about this as I don't work directly on the Hedera Services layer.

[anvabr]

Is there a better place (or individual) I should reach out to with this question? @danielakhterov

[janaakhterov]

A better place might be making an issue/discussion in the services repo https://github.com/hashgraph/hedera-services
A better individual might be @tinker-michaelj which works directly on Hedera services

[anvabr]

Perhaps I am asking a somewhat controversial question here, in the public docs Hedera claims >10k/sec Tx speed so looking at it naively it should take 40 seconds, not 30+ hours. However the published numbers I am sure are for a specific test set-up, I am just trying to figure out if there is a way we can get closer to that architecture, hoping to reduce time significantly.

[janaakhterov]

Unfortunately I don't have much to add there, perhaps something to consider is that throttles are applied to all mint transactions and there might be others minting at the same time you're minting which is why you're seeing slower performance.

[tinker-michaelj]

Hi @anvabr!

Each node on mainnet will accept ~46 NFT mints/sec and each node on testnet will accept ~171 NFT mints/sec. This is because mainnet has 26 nodes and testnet only 7 nodes. Either way, the network capacity for NFT mints works out the same,

26 nodes * 46 NFTs/sec per node ≈ 1200 NFTs/sec
7 nodes * 171 NFT/sec per node ≈ 1200 NFTs/sec

You said that each of your TokenMint operations mint 10 NFTs. So each mainnet node will accept a maximum of ~4.6 such operations per second; and each testnet node up to ~17.1 operations a second.

Now comes the question: How do we use the SDK to submit these operations? If we do something like pattern 1 above,

        const txResponse1 = await signTx1.execute(client);
        const receipt1 = await txResponse1.getReceipt(client);

Then the bottleneck is no longer the 4.6 TPS or 17.1 TPS node throttle; but instead, the time-to-finality of the network. So we want to use something like like your pattern 2, where don't block on the receipt of one TokenMint before submitting the next.

But to avoid surfacing possible SDK issues (or having the nodes close connections to avoid DoS), our goal should be to submit a maximum of 120 such operations per second network wide---since this would saturate the entire network capacity of 1200 NFT mints-per-second.

Does this help provide any color?

[anvabr]

Hello @tinker-michaelj, this helps with understanding somewhat - thank you. It does not fully answer my question of why we are seeing 3 TPS in our NFT minting tests rather than 120, let alone 1200 TPS.

Just guessing here, reading between the lines - are you suggesting we should throttle the submission rate in the testing loop to avoid over-saturating the network?

[anvabr]

There is an interesting report from Dovu here, who have gone to great length to achieve maximum performance for their (non-NFT) use-case. Their experience is that anything above 200 TPS is almost guaranteed to be unstable, and realistic estimate is 40-50 TPS.

[rbair23]

We test on TestNet / MainNet size hosts and networks (26-node network) with equivalent machines -- we really are proving the 10K number on a daily basis. But as Daniel said, not all transactions are throttled to the same levels. The model we use is a "leaky bucket" throttle mechanism that smooths out and averages over a 12 second period -- so it can support spikes. But on public testnet and mainnet you are sharing those throttles with everybody. Each individual submitter doesn't get 10K TPS, it is the aggregate that gets it. And in the case of mint, it is much less than 10K TPS.

[dubgeis]

In order to get to maximum capacity it would be good to attempt to mint and listen in separate threads in the JS SDK and to submit to separate nodes this way. This should lead to some improvement, it also may be something we can set up out of the box if it works so other folks don't run into this issue.

[petreze]

Following the discussion, I just wanted to test the samples provided here against the current stage of the network (since there might be some changes made concerning the exact use case).

So, as mentioned, from the provided example, the first approach is not even a consideration from a performance point of view because you would have to wait to get the receipt of each TokenMint before submitting the next one.

Onto the second approach, running it as it is provided, for a total of 1000 NFT mints ends for roughly 5-6 minutes without any errors. You might see in the console GrpcServiceError: gRPC service failed with status: TIMEOUT error which is basically the equivalent of GrpcServiceError: gRPC service failed with status: UNAVAILABLE which you have encountered previously. This is an enhancement which we made in order to prevent the node going in status unhealthy. Instead, we are trying to submit the transaction to another node in the network so the process does not stop with gRPC status: UNAVAILABLE

Also, I see that you are trying to get the receipt of every TokenMintTx. Would that necessary be your use case because I think this might make it a bit more complicated.

Nonetheless, whenever I set it up to mint 100 000 NFTs, a JS error out of memory FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory (this is both with and without getting the receipt)
This can be fixed if you increase the heap limit when running the file with --max-old-space-size=N tag, where N depends on your local machine's memory

All observations were made on testnet

Please update us on your current intentions if this is still your case
cc: @anvabr @gregscullard @rbair23

[petreze]

Some additional approaches in order to optimize the whole process:

1. Instead of creating a new client for each TokenMintTx and executing the TXs against a random node, you could initialize 7 predefined clients (7 in the case for testnet because there are 7 nodes operating there).

• Each one of the clients will be linked to a specific testnet node with the Client.forTestnet() method.
• Create an array containing all of the clients
• Then, before each TokenMintTx get a random client from the array with something like: nodeClients[Math.floor(Math.random() * nodeClients.length)] and execute the minting with it

Those node IPs and accounts are publicly available here
Thus, the intialization of a client on each for loop iteration should be removed and it would improve the performance a lot

2. Furthermore, you could use a pure Node.js cluster/worker approach:

• where you can get your number of CPUs and fork your cluster in that many workers.
• After that, you add all of the TokenMintTx promises to an array.. let's say promises
• Then you await them await Promise.all(promises) with the help of those N workers which will make the process run across all cpus

cc: @anvabr

[petreze]

As up until now, the best approach from a performance point of view was achieved by using a third party library called bottleneck.js which gives us the opportunity to execute successfully up to ~8000 TokenMintTXs simultaneously. Combined with separating the total amount of transactions (400k) into batches of 5k each, we have achieved 399 979 successful NFT mints for roughly 8 hours and 30 minutes.

Notes: After several days of investigating, the issue with those 21 NFTs missing (each time the number differs) most probably has nothing to do with the network itself or the throttling limitations. Those transactions seem to not reach the application layer in order to be processed correctly. The issue most probably appears during the communication between the SDK and the application layer. This is prioritized and further research is required.

I provide you with a template code piece to try:

import Bottleneck from "bottleneck";
const bottleneck = new Bottleneck();
.....

const clientN =
            nodeClients[Math.floor(Math.random() * nodeClients.length)];

        const seconds = Math.round(Date.now() / 1000);
        const validStart = new Timestamp(seconds, nanosOffset);
        nanosOffset += 1000;
        
        const transactionId = TransactionId.withValidStart(
            operatorId,
            validStart
        );

        const transaction1 = new TokenMintTransaction()
            .setTransactionId(transactionId)
            .setTokenId(tokenId)
            .setTransactionValidDuration(180)
            .setMetadata(data)
            .freezeWith(clientN);
        const signTx1 = await transaction1.sign(privateKey);
        console.log("Execute:", i);


        bottleneck.schedule({
            id: i,
            weight: 1000
        }, async () => {
            console.log('Pushed:', i);
            promises.push(signTx1.execute(clientN));
            return Promise.allSettled(promises);
        });

Note that this code piece is for one batch of transactions only and after the execution of each batch you have to await a bit and clear up the promises list like so:

    await wait(240000);
    promises = [];

cc: @anvabr