[QUESTION] after import event

[dalholm]

Prerequisites

Versions

• PHP version: 7.2
• Laravel version: 5.7
• Package version: 3.1

Description

I have some trouble getting the after import event to work. The event never fires. :(
The after sheet works fine, what am I doing wrong?

i have tried both the register events and the auto register events.

My code

namespace App\Imports;

use App\Order;
use App\Order\Address;
use Illuminate\Contracts\Queue\ShouldQueue;
use Maatwebsite\Excel\Concerns\Importable;
use Maatwebsite\Excel\Concerns\RegistersEventListeners;
use Maatwebsite\Excel\Concerns\ToModel;
use Maatwebsite\Excel\Concerns\WithBatchInserts;
use Maatwebsite\Excel\Concerns\WithChunkReading;
use Maatwebsite\Excel\Concerns\WithCustomCsvSettings;
use Maatwebsite\Excel\Concerns\WithEvents;
use Maatwebsite\Excel\Concerns\WithHeadingRow;
use Maatwebsite\Excel\Events\AfterImport;
use Maatwebsite\Excel\Events\BeforeImport;

class OrderAddressImport implements
    WithHeadingRow,
    ToModel,
    WithBatchInserts,
    WithChunkReading,
    WithCustomCsvSettings,
    ShouldQueue,
    WithEvents
{
    use Importable, RegistersEventListeners;

    protected $order;

    public function __construct(Order $order)
    {
        $this->order = $order;
    }

    /**
     * @param array $row
     * @return Address
     */
    public function model(array $row)
    {
        $data = ['order_id' => $this->order->id];
        $data = array_merge($data, $row);

        return new Address($data);
    }


    public function batchSize() : int
    {
        return 1000;
    }

    public function chunkSize() : int
    {
        return 5000;
    }

    /**
     * @return array
     */
    public function getCsvSettings(): array
    {
        return [
            'delimiter' => ';',
            'enclosure' => '',
            'input_encoding' => 'UTF-8'
        ];
    }

    public static function afterImport(AfterImport $event)
    {
       dd($event);
    }
}

[joaoprado]

I have the same issue. It seems the AfterImport event is never fired if you're implementing the WithChunkReading contract.

[jgodish]

This appears to also be true for the BeforeImport event

[dalholm]

I had a quick look at the code and can not see any code that will fire this event at all in the ChunkReader class.

And when i think about it there is no nice solution for this. Maybe one solution could be to ad a single job to the chain for it to raise the afterImport method?

[dalholm]

Do we have any answers to this topic? :)

[GlennM]

Yes, we've identified this as a bug. We'll see if we can come up with a solution for it in a future version.

[SeoRoman]

@GlennM Speaking with this, I would like to see if there is a way to also implement a BeforeChunk and AfterChunk event as well, to be able to update a front end gui of progress as it is worked... Things to consider with this, should be total count + total chunks + current chunk (just for thoughts), most of my files are done through the GUI but are fairly large, so being able to give them some kind of "it's in progress" is better than an indeterminate progress bar

[patrickbrouwers]

BeforeChunk and AfterChunk sound possible.

The BeforeImport/AfterImport are more tricky. Adding them as 2 extra jobs to the chain, will have performance impact, as it will 2x times re-open the file. The best option would be able to only execute the beforeimport on the first job and the afterimport on the last job. I'll see if that's possible.

[sgilberg]

+1 to adding BeforeImport/AfterImport events to chunk jobs. I was about to submit a proposal to add these as two jobs to the chain, but it sounds like you're already on it! Your suggestion of implementing the events on the first and last job sounds best too if possible.

[alissonalberini]

AfterImport does not work with ShouldQueue and WithChunkReading.
In addition, they are required for queues, so an event at the end of running a queue is not implemented.

with the suggestion of documentation:

$ import-> queue ('users.xlsx') -> chain ([
new NotifyUserOfCompletedImport (request () -> user ()),
]);

Each of the queue items is fired the: NotifyUserOfCompletedImport

Alguma sugestão?

[kliment-toshkov]

I've run into the same issue. Solved it by creating a Job implementing ShouldQueue, then running the Import from it and performing the necessary actions (that otherwise would be in afterImport event) after the Import is finished. The Import has configured batch inserts and chunk reading and is executed synchronously in the Job.

[kliment-toshkov]

One benefit of the above is that I can easily control which Queue worker is executing the job.

[kliment-toshkov]

and another (very important) benefit is that I can import XLS files with the queued job. Something that is not possible if using the standard approach with implementing ShouldQueue in the Import class itself.

[zxl2006001]

ihave the same question。。。how can i do ....please

[dalholm]

@zxl2006001 With the updated documentation from @patrickbrouwers i solved my issue with this logic.

(new UsersImport)->queue('users.xlsx')->chain([
    new NotifyUserOfCompletedImport(request()->user()),
]); 

see here: https://docs.laravel-excel.com/3.1/imports/queued.html#appending-jobs

[PawelDmochowski]

There are actually two more consequences of current design:

1. Looking at Reader classes read method it's not obvious why beforeImport is fired only inside beforeReading instead of being fired synchronously before ChunkReader reads are invoked.

2. In chunked scenario you can chain a job to do any notification stuff but since "currentFile" property is protected you cannot do proper cleanup of temporary file and it's not done automatically - a job chained after all ReadChunk's would be very helpful both with afterImport and with temp cleanup.

[ldiebold]

Do we have any workarounds for handling failed imports? The only method I can currently think of is using laravels global job error event.

Problem with that is I have no control over the aatwebsite\Excel\Jobs\ReadChunk process. This means I can't pass the global event the data I need about the failed job.

How can we handle errors when using chunkSize?

[patrickbrouwers]

Next release (currently 3.1-dev) will have support for Before/AfterImport events for chunk reading. It will also add ImportFailed event that gets raised when any of the chunk jobs fail. (https://twitter.com/patrickbrouwers/status/1114166252711415808)

Feel free to test it on the 3.1 branch and let me know if it works like excepted.

[glennkarlsen]

If I use database queue driver for imports and implement WithChunkReading on my import class it seams to trigger AfterImport event multiple times. Is that expected behavior?

[khantilpatel]

I'm experiencing AfterImport being triggered twice after every successful import, with database queue driver and version 3.1.

[makproductions]

Im experiencing AfterImport being triggered indefinitely after running a successful import also with database queue driver and version 3.1

[sineld]

If I use redis queue driver for imports and implement WithChunkReading on my import class it seams to trigger AfterImport event multiple times. Is that expected behavior?

Here is error log:

local.ERROR: unlink(/Users/sineld/Code/project/public/uploads/tmp/laravel-excel-qfK6O5aCHxhFXioVayQbLQag5UnFKyxB.xlsx): No such file or directory {"exception":"[object] (ErrorException(code: 0):

[adriaanzon]

I'm also running into AfterImport listeners being called twice. After debugging with xdebug I found that the listeners are registered in two places:

• https://github.com/Maatwebsite/Laravel-Excel/blob/d0231ab1f4bb93c8695630cb445ada1fdc54add0/src/Sheet.php#L232-L234
• https://github.com/Maatwebsite/Laravel-Excel/blob/d0231ab1f4bb93c8695630cb445ada1fdc54add0/src/Reader.php#L388-L390

[patrickbrouwers]

Fixed for next release

[elchroy]

How can I notify a user of failed validation errors in a job? The importFailed is a static method, so I do not have access to $this->failures(), and I used the SkipsFailures trait to grab all the validation errors?

Currently, I dispatched a separate job in my collection method, and that job handles the notification, but I think it is more intuitive to use one of the events. Also, I used the AfterImport event to send the notification, but that meant sending two notifications; one for failed rows, and one for successful imports.

Please share if there is a better way to achieve this, and send one notification containing both successful and failed rows? I can only think of using the ImportFailed event, but as I mentioned, it is a static method. Thanks 🙏🏼

[elchroy]

This section suggests using a closure to handle failed validation errors while queuing an import operation.

And another section discourages registering the event handlers as Closures, because of PHP's concerns with serializing closures.

I'd appreciate any advice on how to achieve this. Thanks again

[aavinseth]

how do I access $this in afterImport function? I want to send mail with some dynamically generated data

[sgilberg]

Hi @aavinseth I achieved this using $event->getConcernable() as in the example below:

// ...
public function __construct($import_id)
{
    $this->import_id = $import_id;
}
// ...
public static function afterImport(AfterImport $event)
{
    $importer = $event->getConcernable(); // This class
    $import_id = $importer->import_id; // Access class properties
    // Whatever else you need to do
}

[sgilberg]

And since I see the @elchroy question is still open, there are multiple ways to register event listeners, and closure is only one. So if you are using model serialization, you probably want to use a different method. See https://docs.laravel-excel.com/3.1/imports/extending.html

[farshadff]

This section suggests using a closure to handle failed validation errors while queuing an import operation.

And another section discourages registering the event handlers as Closures, because of PHP's concerns with serializing closures.

I'd appreciate any advice on how to achieve this. Thanks again

hi , did you find any way to solve this issue ? i have the same problem as you have . i would be thankful if you give some advice if you have solved that

[aavinseth]

This section suggests using a closure to handle failed validation errors while queuing an import operation.
And another section discourages registering the event handlers as Closures, because of PHP's concerns with serializing closures.
I'd appreciate any advice on how to achieve this. Thanks again

hi , did you find any way to solve this issue ? i have the same problem as you have . i would be thankful if you give some advice if you have solved that

I have use the database, I am saving the data in every chunks in db with uuid and in afterImport I am processing that data like sending mail, etc... and delete it at last

[farshadff]

This section suggests using a closure to handle failed validation errors while queuing an import operation.
And another section discourages registering the event handlers as Closures, because of PHP's concerns with serializing closures.
I'd appreciate any advice on how to achieve this. Thanks again

hi , did you find any way to solve this issue ? i have the same problem as you have . i would be thankful if you give some advice if you have solved that

I have use the database, I am saving the data in every chunks in db with uuid and in afterImport I am processing that data like sending mail, etc... and delete it at last

thanks for the answer but i didnt get how you have gathered the failures ? because for database store you can only get the success imports

[aavinseth]

This section suggests using a closure to handle failed validation errors while queuing an import operation.
And another section discourages registering the event handlers as Closures, because of PHP's concerns with serializing closures.
I'd appreciate any advice on how to achieve this. Thanks again

hi , did you find any way to solve this issue ? i have the same problem as you have . i would be thankful if you give some advice if you have solved that

I have use the database, I am saving the data in every chunks in db with uuid and in afterImport I am processing that data like sending mail, etc... and delete it at last

thanks for the answer but i didnt get how you have gathered the failures ? because for database store you can only get the success imports

Insert into db whenever your condition or validator fail and process it in afterImport function

[sgilberg]

@farshadff Building off of @aavinseth's answer with a bit more detail: It sounds like you might be struggling with how to use the event listeners without a closure? As I mentioned in my comment last July, there are multiple ways to register event listeners. I used a different method mentioned in the Extending section of the docs. If you combine this with the SkipsOnFailure concern, you can use the onFailure() method to aggregate the failures found in each row, and then use the afterImport event to process the failures at the end. Here's an example based on one of mine:

namespace App\Imports;

use Maatwebsite\Excel\Concerns\Importable;
use Maatwebsite\Excel\Concerns\RegistersEventListeners;
use Maatwebsite\Excel\Concerns\SkipsOnFailure;
use Maatwebsite\Excel\Concerns\WithEvents;
use Maatwebsite\Excel\Concerns\WithValidation;
use Maatwebsite\Excel\Events\AfterImport;
use Maatwebsite\Excel\Validators\Failure;

class MyImport implements SkipsOnFailure, WithEvents, WithValidation
{
    use Importable, RegistersEventListeners;

    public function __construct($import_id)
    {
        $this->import_id = $import_id;
    }

    // ...

    public function onFailure(Failure ...$failures)
    {
        // gather and save your failures as you go (I chose to cap mine at 500)
        $import = $this->getImport($this->import_id); // your function to get the import model
        if (!$import->validation_failures || count($import->validation_failures) < 500) {
            $failure_count = $import->validation_failures ? count($import->validation_failures) : 0;
            $validation_failures = [];
            foreach ($failures as $failure) {
                $failure_count++;
                // Empty row, so we skip.
                if (empty(array_filter($failure->values()))) {
                    continue;
                }

                $validation_failures[] = [
                    'row' => $failure->row(),
                    'attribute' => $failure->attribute(),
                    'errors' => $failure->errors(),
                ];
                if ($failure_count >= 500) {
                    break;
                }
            }

            $import->validation_failures = array_merge((array) $import->validation_failures, $validation_failures);
            $import->save();
        }
    }
    
    public static function afterImport(AfterImport $event)
    {
        // process the failures at the end (send any notifications, etc)
        $importer = $event->getConcernable(); // This class
        $import_id = $importer->import_id;
        // etc
    }
}