Solidity: Incorrect code count from misreading valid comments

[alexroan]

Running tokei on the solidity files in this repo returns incorrect code counts. The following example is from analyzing src/protocol/ThunderLoan.sol.

This code snippet appears at the top of the file.

...

// SPDX-License-Identifier: AGPL-3.0-only
pragma solidity 0.8.20;

import { SafeERC20 } from "@openzeppelin/contracts/token/ERC20/utils/SafeERC20.sol";
import { AssetToken } from "./AssetToken.sol";
import { IERC20 } from "@openzeppelin/contracts/token/ERC20/IERC20.sol";
import { IERC20Metadata } from "@openzeppelin/contracts/token/ERC20/extensions/IERC20Metadata.sol";
import { OwnableUpgradeable } from "@openzeppelin/contracts-upgradeable/access/OwnableUpgradeable.sol";
import { Initializable } from "@openzeppelin/contracts-upgradeable/proxy/utils/Initializable.sol";
import { UUPSUpgradeable } from "@openzeppelin/contracts-upgradeable/proxy/utils/UUPSUpgradeable.sol";
import { OracleUpgradeable } from "./OracleUpgradeable.sol";
import { Address } from "@openzeppelin/contracts/utils/Address.sol";
import { IFlashLoanReceiver } from "../interfaces/IFlashLoanReceiver.sol";

contract ThunderLoan is Initializable, OwnableUpgradeable, UUPSUpgradeable, OracleUpgradeable {
    error ThunderLoan__NotAllowedToken(IERC20 token);
    error ThunderLoan__CantBeZero();
    error ThunderLoan__NotPaidBack(uint256 expectedEndingBalance, uint256 endingBalance);
    error ThunderLoan__NotEnoughTokenBalance(uint256 startingBalance, uint256 amount);
    error ThunderLoan__CallerIsNotContract();
    error ThunderLoan__AlreadyAllowed();
    error ThunderLoan__ExhangeRateCanOnlyIncrease();
    error ThunderLoan__NotCurrentlyFlashLoaning();
    error ThunderLoan__BadNewFee();

    using SafeERC20 for IERC20;
    using Address for address;

    /*//////////////////////////////////////////////////////////////
                            STATE VARIABLES
    //////////////////////////////////////////////////////////////*/
    mapping(IERC20 => AssetToken) public s_tokenToAssetToken;

    // The fee in WEI, it should have 18 decimals. Each flash loan takes a flat fee of the token price.
    uint256 private s_feePrecision;
    uint256 private s_flashLoanFee; // 0.3% ETH fee

.... (more solidity)

Tokei is able to count lines up until this block:

    /*//////////////////////////////////////////////////////////////
                            STATE VARIABLES
    //////////////////////////////////////////////////////////////*/

Resulting in an incorrect code count of 23 (essentially all the code lines before this comment syntax). This indicates that this particular valid solidity comment is breaking the parsing somehow. The expected count is over 100.

[TilakMaddy]

Just to make it easy for the maintainer - here's my observations:

Consider this:

pragma solidity >=0.4.22 <0.6.0;
/* 
    BLAH
/*/
contract Foo {
    function foo(address bar) public pure {
         require(bar != address(0));
    }
}

It spits out "1 code line, 8 comments"

Problem is Line 4:

It is having trouble deciding whether to consider the first two chars /* and call it the starting of the multi-line comment or to consider the last two characters */ and call it the ending of multi-line comment.

As of now, tokei does the former.

Possible fix (take it w/ a grain of salt): If we know we are in the multi-line comment stack, why not aggressively "eat" characters until we find the closing? I'm sure it's not as trivial a change because languages like c++ have define macros and it shouldn't be the responsibility of tokei to know where your comment actually starts and where it ends.