Total in-transaction crypto value < total taxable crypto value

[nicksettoon]

Hello! I've graduated to working with rp2, finally.

I'm getting an issue when running rp2 on the combined dataset from all my exchanges (coinbase, coinbasepro, bittrex, and binance, excluding DEXes for now). I'm not sure how "total taxable crypto value" is being calculated here, but I've verified all the data is in the correct format and each transaction type on each exchange has the correct data. I've excluded all crypto transfers (deposits/receives and withdraws/sends) and marked all trades as having occurred on the same exchange (binance). I've also populated spot price for all transactions "manually" through the coingecko API, so dali shouldn't be missing spot price when building the input file.

I've got some stuff i traded on DEXes (and some dead exchanges) that isn't (or can't be) included. However, those trades were way after the date of the transaction I'm getting this error on (2017-12-28).

I'm going through looking for negative running balances, but I've been getting this error no matter what I do so i figured i'd ask for insight.

2022-10-10 18:51:41,010/rp2/INFO: Country: us
2022-10-10 18:51:41,011/rp2/INFO: Generation Language: en
2022-10-10 18:51:41,012/rp2/INFO: Accounting method: hifo
2022-10-10 18:51:41,012/rp2/INFO: Configuration file: output/test_crypto_data.ini
2022-10-10 18:51:41,012/rp2/INFO: Input file: output/test_crypto_data.ods
2022-10-10 18:52:03,112/rp2/INFO: Processing ABY
2022-10-10 18:52:05,814/rp2/ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/home/vagrant/.local/lib/python3.10/site-packages/rp2/rp2_main.py", line 141, in _rp2_main_internal
    computed_data: ComputedData = compute_tax(configuration=configuration, accounting_engine=accounting_engine, input_data=input_data)
  File "/home/vagrant/.local/lib/python3.10/site-packages/rp2/tax_engine.py", line 39, in compute_tax
    unfiltered_gain_loss_set: GainLossSet = _create_unfiltered_gain_and_loss_set(configuration, accounting_engine, input_data, unfiltered_taxable_event_set)
  File "/home/vagrant/.local/lib/python3.10/site-packages/rp2/tax_engine.py", line 152, in _create_unfiltered_gain_and_loss_set
    raise RP2ValueError("Total in-transaction crypto value < total taxable crypto value") from None
rp2.rp2_error.RP2ValueError: Total in-transaction crypto value < total taxable crypto value

[eprbell]

The error is saying that for token ABY, at a certain point in time the total amount in is less than the total amount that is taxed: e.g. this can happen if you buy 1 ABY and later sell 1.5 ABY. For an example of this error and its related discussion check #24.

How many ABY transactions do you have? I suggest trying to repro with a simple case (i.e. remove ABY transactions while still making sure the error occurs). If you have a small enough use case I can take a look.

[nicksettoon]

I've gotta do a better job of searching github. In my defense, the knowledge around here is pretty spread out.
Still tho, now that I understand the error I see it's obviously because of the fee in the buy transaction. The fee was being subtracted from the crypto_in amount, leaving not enough for the sell transaction. That's on me formatting the data i put into dali wrong and leaving fees for those types of transactions in.

In my travels I changed the exchange value for these transactions and also got rp2.rp2_error.RP2ValueError: Parameter 'exchange' value is not known: bittrex. Is there a list of exchanges recognized by rp2? If it's just the ccxt list, 'bittrex' is their id for that exchange source. this was because the rp2 .ini file didn't have bittrex in the list of exchanges. (duh)

I also got rp2.rp2_error.RP2ValueError: ADA OutTransaction (2018-01-10 07:28:00+00:00, id 362): fee-typed transaction has non-zero 'crypto_out_no_fee', but i also think this is me formatting the input data wrong. I thought when I broke out a transaction into it's own FEE transaction that "crypto_out_no_fee" is the logical column for that data to be in. turns out i don't need all these separate FEE transactions at all. i'm just confusing myself.

I'm still confused about transforming all these crypto-crypto transactions from the exchange representation to dali's representation for so many line items. I thought I understood it, but i guess not. just need to put more time in.

[eprbell]

It's all good! Yours is a valid user experience and it provides valuable information: e.g. the error you got is indeed a bit cryptic and I need to make it a little easier to consume (you're not the only user who tripped on it).

The list of supported exchanges is at: https://github.com/eprbell/dali-rp2/blob/main/docs/configuration_file.md#data-loader-plugin-sections. However it will be updated with all the latest plugins in development once eprbell/dali-rp2#90 is merged (soon).

Thanks for the feedback!

[eprbell]

Strange: are you sure sum sells are done using crypto_out_with_fee? Can you create a BTC-only small repro? If so I can take a look.

[nicksettoon]

crypto_out_with_fee isn't populated. I uploaded just btc and gave you access.

Process

remove the in/out separator lines, then sort by time stamp

balance change in O column
=IF(OR(E3="Buy",E3="Gift",E3="Income",E3="Staking"),H3-I3,-H3-I3)

running balance in P column
=SUM($O$3:O3)

negative check in Q column
=IF(P3<0)

search for TRUE

[eprbell]

Can you upload the config file as well? The ODS file is very large: I'll take a quick look but I won't have the time to dig deep and find a problem in a file with a lot of transactions. However if you can reduce it to a few transactions it will be more likely that I can find what's wrong.

[nicksettoon]

not sure which transactions are the ones causing the issue, so idk how to reduce without removing the phenomena.

[eprbell]

See my comment below.

[eprbell]

I just noticed the ODS you uploaded is not well-formed: the IN table has Sell and Fee-typed transactions (which should belong in the OUT table). Without the config file I can't run it, but I suspect it would cause parse errors. Please upload a well-formed ODS file and its respective config file, so I can repro.

[nicksettoon]

lol. sorry. done. idk what i was thinking. been a long week.

[eprbell]

I reduced your input file from 15800 to 600 lines and uploaded it to your private prj: still large but a little more manageable. The out transaction where the error occurs is the one with timestamp 2018-01-10 01:22:22-0600.

I won't have much time to work on this in the next 10 days, here's an investigation idea, if you want to pursue it:

• use the modified 600-line file as input
• use FIFO as the accounting method
• instrument tax_engine.py:_create_unfiltered_gain_and_loss_set() as follows:
  • before the while loop print the following: taxable_event, taxable_event_amount, acquired_lot, acquired_lot_amount
  • inside the while loop there are 4 if branches. At the end of each branch print the following: taxable_event, taxable_event_amount, acquired_lot, acquired_lot_amount

Then run and capture the output: you should be able to follow how output lots (taxable events) are mapped to input lots (acquired lots), including how they are fractioned.

[nicksettoon]

thanks will look into it

[eprbell]

Found the problem: it's a subtle RP2 bug, a corner case. I will try to fix it in the next 1-2 days and will provide full details. Stay tuned and thanks for reporting!

[eprbell]

Here is a summary of the fix for this issue. Your data exposed two subtle bugs in RP2:

• the accounting engine keeps acquired lots in an AVL tree, using a concatenation of timestamp and internal id as key. This allows for easy lookup of acquired keys based on timestamp. The bug was that the key was being constructed without normalizing timezone first, which caused a mis-ordering of non-UTC timestamps: in turn this generated relatively rare and fairly obscure bugs, like the one described in this discussion.
• the tax_report_us output plugin resizes its various output sheets with a number of rows that is proportional to the number of transactions (in the gain loss set) that have type compatible with the given sheet. Normally this is a 1-1 relationship (e.g. the Capital Gains sheet only has Sell-typed transactions, the Airdrop sheet only has Airdrop-typed transactions, etc.). However the Investment Expenses sheet can contain transactions of multiple types (Move, Fee). However the code was only adding rows for Move transactions and not for Fee. So if a data set had enough Fee-typed transactions (like in your case), the Investment Expense sheet would run out of rows and the plugin would throw an exception.

I have fixes for both these bugs: I'm testing the code and I expect to merge soon.

I also added instrumentation in tax_engine.py, to make it easier to find out what is happening when a user gets the "Total in-transaction crypto value < total taxable crypto value" error. In this case the error was caused by a RP2 bug, but normally it denotes a problem in the user data, which used to be relatively hard for users to resolve. The new instrumentation should make it much easier for users to fix this problem in their data. It is activated with LOG_LEVEL=DEBUG and it logs the internal progress of the tax engine when it uses the accounting engine to pair lot fractions.

In summary, this error report led to 3 neat improvements: 2 bug fixes and 1 logging extension which will make user data errors easier to fix in the future.

Fixes are coming soon, stay tuned.

[eprbell]

The bug has been fixed in the latest versions of RP2 (1.3.1) and DaLI (0.6.1): see above for description of both bug and fix. Thanks again for reporting!

[nicksettoon]

Wow! No, thank you. I really appreciate the work you've done here. Hopefully I won't have any more issues.

[eprbell]

I forgot to mention: I created a tiny repro for this bug (3 transactions) and incorporated it into the automated testing framework so that we can be sure this doesn't regress. It's in test_data4.ods, sheet B3.