Feature: inbound email expenses #9042

paulwer · 2023-12-14T07:04:44Z

this PR is a draft for importing inbound emails as expenses.
I would like to use this draft to may ask some questions and to develop this feature.

PostMark InboundWebhook Processing: https://postmarkapp.com/developer/webhooks/inbound-webhook
Mailgun https://documentation.mailgun.com/en/latest/quickstart-receiving.html
Brevo https://www.brevo.com/inbound-parsing/
waiting for: Feature: Brevo Email Provider #9053

Any more providers to cover?

Status (Tests):

Dev Environment

Webhook Relay on Localhost
I have used webhook relay to simulate a internet contactable url to test the webhook behavior with each provider.

Documentation

Company Configuration

If you are starting with the IngresEngine, you should have to take a look at some major points. First of all the mailbox recieving the email has to be allowed, which can be configured at the company level. The IngresEngine can denie incomming emails, when the configuration and data related to your vendors allows it to.

expense_mailbox_active Enable Disable Mail Processing
expense_mailbox E-Mail of the recieving mailbox
inbound_mailbox_allow_company_users If true, all of your users of your own company are allowed to send in emails, which will not get blocked
inbound_mailbox_allow_vendors If true and the sender of an email matches the vendor inbound mailbox configuration it will not get blocked
inbound_mailbox_allow_unknown If true all mails will not get blocked
inbound_mailbox_whitelist a comma seperated list of additional domains or full emails, which should be allowed to send emails (even when expense_mailbox_allow_unknown is false)
inbound_mailbox_blacklist a comma seperated list of blocked domains or full emails, which should not be allowed to send emails (even when expense_mailbox_allow_unknown is true or the mail would be allowed by any other setting)

Vendor Matching

Incomming emails are beeing attempted to match against a vendor by checking the vendor contact emails.

SetUp Inbound Providers

SetUp Mailgun

To ensure delivery, you have to configure all used mailgun regions with a recieving configuration using store & notify.

Create and setup your mailgun account
Prepare your DNS settings for inbound mail processing, by setting up necessary MX records
Create a recieving route using store and notify
3a. Go to Recieving Section and click on "Create Route"

3b. Create your route by selecting your mailbox configured in invoiceninja and add notify endpoint /api/v1/mailgun_inbound_webhook

SetUp Brevo

To ensure delivery you have to configure brevo inbound parsing according to: https://developers.brevo.com/docs/inbound-parse-webhooks

Create and setup your brevo account
Prepare your DNS settings for inbound mail processing, by setting up necessary MX records
Register the webhook via the brevo api to the following notify endpoint: /api/v1/brevo_inbound_webhook?token=brevo-api-token

SetUp Postmark

To ensure delivery you have to configure brevo inbound parsing according to: https://developers.brevo.com/docs/inbound-parse-webhooks
// TODO

SetUp with Own MailServer

If you want to use your own mailbox, you have to eigher redirect the mail to a mailbox of one of the providers above or use a provider for inbound directly.
We recommend to use one of these providers hosted on a subdomain f.ex. invoicing.xxx.com directly or to use the auto-generated mailbox of postmark when you are using redirect.

app/Jobs/Mail/ExpenseMailboxJob.php

paulwer · 2023-12-15T17:39:31Z

Edited:
Documentation:
company has gained an additional property named expense_mailbox, which is used for all webhook endpoints and can be configured with the following ENV-var:

INBOUND_MAILBOX_ENDINGS: comma seperated list: one of 2 options to validate a mailbox name. Checks if the mailbox ends with one of the strings. should contain the @ symbol.

paulwer · 2023-12-16T15:34:58Z

@turbo124 please check the handling of changing the name of a mailbox and give me feedback on this.

I descided to not use a regex aproach and instead work with checks, if the provided string have a specific ending. Therefore the frontend can create a more easy combination of choosing the ending (dropdown) and let the user descide, which text he/she wants to use for the mailbox (text-input-field).
I also implemented a system-wide check that the mailbox does not exists.

app/Http/ValidationRules/Company/ValidExpenseMailbox.php

app/Factory/CompanyFactory.php

turbo124 · 2023-12-20T06:57:29Z

app/Http/Requests/Company/StoreCompanyRequest.php

@@ -55,6 +56,8 @@ public function rules()
            }


Safe to never check the expenes mailbox in the Store Request as this is configured afterwards.... so only need a check on update

@turbo124 I guess I missed this one :(
Just removing all the checks for the expense_mailbox?

@turbo124 should this not be keep in place, to suppor this, when using api create company?

config/ninja.php

database/migrations/2023_12_10_110951_inbound_mail_parsing.php

turbo124

@paulwer review complete!

…inja into feature-inbound-email-expenses

turbo124 · 2024-05-21T01:48:14Z

app/Http/Controllers/MailgunController.php

+        // @turbo124 TODO: how to check for services.mailgun.webhook_signing_key on company level, when custom credentials are defined
+        // TODO: validation for client mail credentials by recipient


I would imagine here we would need a company.settings.variable to store this?

please explain, what would you like to store here?

edit: sorry i missed my own question. I will create a setting entry for it. do you have any name suggestions or further things to considder?

turbo124 · 2024-05-21T01:49:58Z

app/Http/Controllers/PostMarkController.php

+            return response()->json(['message' => 'Blocked.'], 403);
+        }
+
+        $company = MultiDB::findAndSetDbByExpenseMailbox($input["To"]);


Just to confirm on this one, this particular field would then also require to be unique in the system, ie company.expense_mailbox

yes, thats right :)

can this be resolved or are there any further questions on this topic?

turbo124 · 2024-05-21T01:56:37Z

config/ninja.php

+        'gocardless' => env('GOCARDLESS_KEYS', ''),
+        'square' => env('SQUARE_KEYS', ''),
+        'eway' => env('EWAY_KEYS', ''),
+        'paytrace' => env('PAYTRACE_KEYS', ''),


I think this got removed accidentally, could you add back please?

'mollie', env('MOLLIE_KEYS',''),

oh sorry, it is down below on line 102

i saw, that you have 'mollie', env('MOLLIE_KEYS','') onstead of 'mollie' => env('MOLLIE_KEYS','') in your repo. which one is the right one?

turbo124

Just a few comments and issues to resolve!

…inja into feature-inbound-email-expenses

LarsK1 · 2024-05-21T06:00:47Z

@paulwer I'm planning to finish the e-invoice import by end of the week. So you could integrate by calling (new ZugferdEDocument($string_file))->run(). The class then takes all the heaving processing...

paulwer · 2024-05-21T07:09:45Z

@LarsK1 what happ3ns, when the document has no Zugferd Signature?
In such case the default import, like i implemented it currently should be used.

Any sugestions? Error catching or predetermining if the data required is available?

LarsK1 · 2024-05-21T07:12:51Z

@paulwer The easiest way, would be to first check the file ending (.pdf, .xml), then, if it's a valid ending to pass it over to the ZugferdEDocument class, which validates the file or returns an error, if the file is not valid. For the long term, we will have to create a detection library to determine, which e-invoicing standard the provided document is.

LarsK1 · 2024-05-21T07:19:59Z

I would raise this exception: https://github.com/horstoeko/zugferd/blob/18de152080610d6d77eda97e375bfb8ec03ef4f8/src/ZugferdDocumentReader.php#L230

paulwer · 2024-05-21T07:35:35Z

@turbo124 @LarsK1 my perspective there could be cases, where an xml and a pdf is present. Does anyone of you see a good solution to determine if they are matching.

I see error cases when multiple invoices are present within one email here.

LarsK1 · 2024-05-21T08:07:46Z

@paulwer the first case wouldn't matter. Since the class will ignore duplicate documents (by vendor, invoice number, sum and date matching)
Regarding the second case, all system I know, do handle multiple documents per Mail. They just handle one after another...

paulwer · 2024-05-21T14:37:26Z

@turbo124 @LarsK1 therefore It will stay like currently, that all docuements will be imported as distinct expense. and only those cases which can be matched by Zugferd will result in reuse.

@LarsK1 my solution does saves the email, as well as the document to the expense. Is it possible to pass the email.html document also to your sevice to enable saving this data in your expenses also. / or maybe return the expense, so I can save this document into it

paulwer · 2024-05-22T04:43:44Z

app/Services/InboundMail/InboundMailEngine.php

+        event(new ExpenseWasCreated($expense, $expense->company, Ninja::eventVars(null))); // @turbo124 please check, I copied from API-Controller
+        event('eloquent.created: App\Models\Expense', $expense); // @turbo124 please check, I copied from API-Controller


@turbo124 when creating an expense through the api the following events are thrown:

event(new ExpenseWasCreated($expense, $expense->company, Ninja::eventVars(null))); // @turbo124 please check, I copied from API-Controller event('eloquent.created: App\Models\Expense', $expense); // @turbo124 please check, I copied from API-Controller

should they also be present here?

paulwer · 2024-05-22T04:49:42Z

@LarsK1 regarding multiple documents within one email.
What happens to cases where the sender sends additional informations like a AGB.pdf
They will currently result in an expense with no data just containing this document?

Do you think this should be wanted behavior, because of edge case? Or do you have any other idea :)

LarsK1 · 2024-05-26T05:30:45Z

@paulwer I thought about your idea regarding AGB. Most system have an "inbox" folder, where one can filter the ingested documents. Otherwise, we should have the ability to create "template" expenses, which can be deleted without any remains or accepted and then converted to a "full" expense.

paulwer added 4 commits December 10, 2023 16:06

initial commit

b2b1cce

init inbound webhook

1125f57

env-config

b065542

improv

d8064a9

paulwer commented Dec 14, 2023

View reviewed changes

app/Jobs/Mail/ExpenseMailboxJob.php Outdated Show resolved Hide resolved

paulwer added 10 commits December 14, 2023 10:39

vendor updates

8bd9f34

save mailbox at company

a6f9275

restruct files

8d6925a

removing pop3 integration, becasue we cannot move mails

5adb799

wip: updates

6df2139

wip process normal mailgun webhooks

58a7456

minor comments

dd9727a

validator for ExpenseMailbox property

36279be

switch to endings for better oportunity to display in frontend

168fef7

fixes

0e24ac0

paulwer added 2 commits December 16, 2023 09:17

wip: first initial setup for creating expenses from webhooks

e05db36

handling of temporary files + wip init of webhooks processing

c393fda

paulwer added 4 commits December 16, 2023 16:36

fixes

ad480a5

fix composer

9ef5a25

better ExpenseMailboxJob

8d43eb6

fixes

f0415b6

turbo124 reviewed Dec 16, 2023

View reviewed changes

app/Factory/CompanyFactory.php Outdated Show resolved Hide resolved

paulwer added 5 commits December 18, 2023 15:05

wip: restruct and init IngresEmailEngine

5d70daa

wip: spam protection

7245de8

renaming and minor changes

08662c1

minor changes

5efb33d

ignore emails with no documents attached

450569e

turbo124 reviewed Dec 20, 2023

View reviewed changes

turbo124 reviewed Apr 23, 2024

View reviewed changes

config/ninja.php Outdated Show resolved Hide resolved

turbo124 reviewed Apr 23, 2024

View reviewed changes

database/migrations/2023_12_10_110951_inbound_mail_parsing.php Outdated Show resolved Hide resolved

turbo124 requested changes Apr 23, 2024

View reviewed changes

paulwer added 9 commits April 24, 2024 08:40

replace Log::info with nlog

b2110de

fix ninja.php

5a5df06

fix invalid fields in VendorTransformer

21aa38c

fixes

8378f9c

fixes on MultiDB

96d4a25

fixes: env vars

e08e377

Merge branch 'v5-develop' of https://github.com/invoiceninja/invoicen…

9c9c4a9

…inja into feature-inbound-email-expenses

fix: reduce vars in favor for db

ef60992

fix: env for black and whitelist

22bc942

turbo124 reviewed May 21, 2024

View reviewed changes

turbo124 requested changes May 21, 2024

View reviewed changes

Merge branch 'v5-develop' of https://github.com/invoiceninja/invoicen…

befbdc3

…inja into feature-inbound-email-expenses

paulwer commented May 22, 2024

View reviewed changes

paulwer added 2 commits May 22, 2024 07:07

wip: ZugferdEDocument implementation

8e850ea

minor code cleanups

e204788

paulwer marked this pull request as draft May 24, 2024 14:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: inbound email expenses #9042

Feature: inbound email expenses #9042

paulwer commented Dec 14, 2023 •

edited

paulwer commented Dec 15, 2023 •

edited

paulwer commented Dec 16, 2023 •

edited

turbo124 Dec 20, 2023

paulwer Apr 4, 2024 •

edited

paulwer Apr 7, 2024

turbo124 left a comment

turbo124 May 21, 2024

paulwer May 21, 2024 •

edited

turbo124 May 21, 2024

paulwer May 21, 2024

paulwer May 21, 2024

turbo124 May 21, 2024

paulwer May 21, 2024

paulwer May 21, 2024

turbo124 left a comment

LarsK1 commented May 21, 2024

paulwer commented May 21, 2024

LarsK1 commented May 21, 2024

LarsK1 commented May 21, 2024

paulwer commented May 21, 2024

LarsK1 commented May 21, 2024

paulwer commented May 21, 2024

paulwer May 22, 2024 •

edited

paulwer commented May 22, 2024

LarsK1 commented May 26, 2024

		// @turbo124 TODO: how to check for services.mailgun.webhook_signing_key on company level, when custom credentials are defined
		// TODO: validation for client mail credentials by recipient

		event(new ExpenseWasCreated($expense, $expense->company, Ninja::eventVars(null))); // @turbo124 please check, I copied from API-Controller
		event('eloquent.created: App\Models\Expense', $expense); // @turbo124 please check, I copied from API-Controller

Feature: inbound email expenses #9042

Are you sure you want to change the base?

Feature: inbound email expenses #9042

Conversation

paulwer commented Dec 14, 2023 • edited

Dev Environment

Documentation

Company Configuration

Vendor Matching

SetUp Inbound Providers

SetUp Mailgun

SetUp Brevo

SetUp Postmark

SetUp with Own MailServer

paulwer commented Dec 15, 2023 • edited

paulwer commented Dec 16, 2023 • edited

Choose a reason for hiding this comment

paulwer Apr 4, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turbo124 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulwer May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turbo124 left a comment

Choose a reason for hiding this comment

LarsK1 commented May 21, 2024

paulwer commented May 21, 2024

LarsK1 commented May 21, 2024

LarsK1 commented May 21, 2024

paulwer commented May 21, 2024

LarsK1 commented May 21, 2024

paulwer commented May 21, 2024

paulwer May 22, 2024 • edited

Choose a reason for hiding this comment

paulwer commented May 22, 2024

LarsK1 commented May 26, 2024

paulwer commented Dec 14, 2023 •

edited

paulwer commented Dec 15, 2023 •

edited

paulwer commented Dec 16, 2023 •

edited

paulwer Apr 4, 2024 •

edited

paulwer May 21, 2024 •

edited

paulwer May 22, 2024 •

edited