Support for back dated transactions

[nicoabie]

Is your feature request related to a problem? Please describe.
In some scenarios it is needed to create transactions that happened in the past and not in the moment they were committed.

Summary

Solution proposal
Make the api accept a backdate property

{
  "postings": [
    {
      "source": "alice",
      "destination": "teller",
      "amount": 100,
      "asset": "COIN"
    },
    {
      "source": "teller",
      "destination": "alice",
      "amount": 5,
      "asset": "GEM"
    }
  ],
  backdate: $timestamp
}

This should update the balances accordingly or throw an error if subsequent transactions cannot be applied to avoid leaving the system in an invalid state

[gfyrag]

Hi @nicoabie !
We have discussions about this feature.
It is on our roadmap :)

[nicoabie]

Hi @gfyrag thanks for answering!
I didn't find that in the roadmap so I created the ticket. Is there any time estimate? Is there any way to contribute in its development?

[nicoabie]

Pinging @altitude

We've been talking about this on slack with @wyatt-rematter and Geoffrey

We believe it could be implemented in the following manner:

If the transaction is positive then we need to adjust the following balances (the easy case)

If the transaction is negative we need to make sure that the balance is always bigger than the number we want to subtract in the backdated transaction.

In order to do that a first very cheap check would be to verify the latest balance. If it is smaller, then the transaction cannot be applied.

If it is larger then we would need to check if all the previous balances are larger than the amount we want to subtract.

If false we cannot apply the transaction

if true we apply the transaction and decrease all the subsequent balances.

What are your thoughts?

[hurrycaine]

Warning Formance user noob here so take care. ;o)

I just been doing a spike and I raised concerns about user generated timestamp on create transactions.
I was worried about the timestamp generated from different machines.
With the way the time is floored when used in Formance, a very small difference could cause the system to think this is a back dated transaction and throw an error. Even if the timestamp was rounded, the issue could very easily happen. I really do not see how it (external timestamp generation) is usable in any complex system.

I was pointed to this discussion which is a bit different but this just got me thinking more.
By allowing a user generated timestamp you can change history (if not for the validation check) and even set transactions in the future, which would break any normal transaction afterwards.

Letting Formance set the timestamp seems best to me, which could be thought of an insert time (to be fair it's never called a created_at timestamp). Any other time could be added to meta-data. The timestamp in the meta-data may be important for reporting for example. So really the external transaction create time is a separate concern to the insert time into Formance.

So circling back to this discussion - I am curious to know what scenarios would it be best to backdate a transaction?
Would all affected transactions PreCommitVolume and PostCommitVolumes have to be re-calculated?

If you look at all the checks listed above - why not tack the transaction at the current place in the Ledger and have the back date in the meta-data?

If Formance is reporting out balances and there was a NSF on a withdrawal due to a missed deposit. That really happened, the missed deposit transaction should be fixed but tacked to the end. The balances are now fixed, you also have a history on why the withdrawal was not allowed.

If there was no failed transaction just balances not correct on a given date. I think again - this should be meta-data driven not an insert into a (Formance) database.

Thoughts?

[nicoabie]

Hey @hurrycaine!
Very interesting points.

There are systems that allow back dating transactions and it is due the fact that users not always have all the information at hand to fill it out. Or they are lazy and enter it a few days after.

I agree that there must be a timestamp generated by formance to indicate when the transaction was committed to the ledger but that timestamp does not necessarily needs to mean that the transaction took place at that particular moment in time.

We are building a system for recyclers. They but material at certain price every day, it may happen that they do not input the transaction on the same day but in the next few days. For us to calculate the average acquisition cost it is important to know when the transaction took place and not when it was entered into the ledger.

Pre and Post commit volumes need to be recalculated yes and there should be a validation to make sure the back date is in the past.

[walmor-rematter]

Hey @gfyrag! I work with Nico, we're starting the implementation of a module in our system that could potentially use Formance, but the support for back-dated transactions is a must for us. Is this still on your roadmap? Will this be included in version 2?