Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DataPrivacy Task fails in some situations #3402

Closed
MrGeneration opened this issue Feb 9, 2021 · 8 comments
Closed

DataPrivacy Task fails in some situations #3402

MrGeneration opened this issue Feb 9, 2021 · 8 comments
Labels
bug data privacy prioritised by payment verified

Comments

@MrGeneration
Copy link
Member

MrGeneration commented Feb 9, 2021 •
edited

Infos:

  • Used Zammad version: 3.6
  • Installation method (source, package, ..): any
  • Operating system: any
  • Database + version: any
  • Elasticsearch version: any
  • Browser + version: any
  • Note: Stumbled upon this during Monitoring tasks on Hosted setups. Happens more often recently.

Expected behavior:

DataPrivacy will adjust or remove all references of to be deleted users if needed.

Actual behavior:

When deleting a agent, it sometimes happens that those tasks fail because of the the user still being referenced within ActivityStreams.

After clearing those issues, you usually can reprocess the DataPrivacy Task without issues.

Example:

#<ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  update or delete on table "users" violates foreign key constraint "fk_rails_5e0981116b" on table "activity_streams"
DETAIL:  Key (id)=(4) is still referenced from table "activity_streams".
: DELETE FROM "users" WHERE "users"."id" = $1>
DataPrivacyTask.find(1) notify UPDATED 2021-02-09 10:19:55 UTC
Found no ActiveJobLock for DataPrivacyTaskJob (Job ID: d2e8f8c9-305a-417f-8d4d-3925f9bf823b) with key 'DataPrivacyTaskJob'.
Performed DataPrivacyTaskJob (Job ID: d2e8f8c9-305a-417f-8d4d-3925f9bf823b) from DelayedJob(default) in 497.72ms
 => nil

Steps to reproduce the behavior:

  • work as to be deleted agent
  • let someone delete the account after wards

Yes I'm sure this is a bug and no feature request or a general question.
@mantas
Copy link
Collaborator

mantas commented Feb 9, 2021

user is still referenced within ActivityStreams

As in user's own activity stream or referenced in other user's activity stream?

@MrGeneration
Copy link
Member Author

As in user's own activity stream or referenced in other user's activity stream?

I can't answer that question.
Currently I have to remove the complete ActivityStream, because it's not enough to clear o_id: <affected-user> and created_by_id: <affected-user>.

@rolfschmidt
Copy link
Collaborator

Next time this errors occur we could check what column is the relation to the error via psql console \d+ table (checking the foreign key name).

We also could add a dumper to the app/models/concerns/has_activity_stream_log.rb to see if we mistakenly create new entries for the activity stream log while we deleting the user.

@MrGeneration
Copy link
Member Author

MrGeneration commented Feb 12, 2021 •
edited

So this affects an instance again.
This time I ensured to look deeper.

The affected user is an agent.
The affected user has ID 235017.

#<ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR:  update or delete on table "users" violates foreign key constraint "fk_rails_5e0981116b" on table "activity_streams"
DETAIL:  Key (id)=(235017) is still referenced from table "activity_streams".
: DELETE FROM "users" WHERE "users"."id" = $1>

So I then checked the references - here's a complete dump:

=> \d+ activity_streams
                                                                  Table "public.activity_streams"
          Column           |              Type              |                           Modifiers                           | Storage | Stats target | Description
---------------------------+--------------------------------+---------------------------------------------------------------+---------+--------------+-------------
 id                        | integer                        | not null default nextval('activity_streams_id_seq'::regclass) | plain   |              |
 activity_stream_type_id   | integer                        | not null                                                      | plain   |              |
 activity_stream_object_id | integer                        | not null                                                      | plain   |              |
 permission_id             | integer                        |                                                               | plain   |              |
 group_id                  | integer                        |                                                               | plain   |              |
 o_id                      | integer                        | not null                                                      | plain   |              |
 created_by_id             | integer                        | not null                                                      | plain   |              |
 created_at                | timestamp without time zone    | not null                                                      | plain   |              |
 updated_at                | timestamp(3) without time zone | not null                                                      | plain   |              |
Indexes:
    "activity_streams_pkey" PRIMARY KEY, btree (id)
    "index_activity_streams_on_activity_stream_object_id" btree (activity_stream_object_id)
    "index_activity_streams_on_activity_stream_type_id" btree (activity_stream_type_id)
    "index_activity_streams_on_created_at" btree (created_at)
    "index_activity_streams_on_created_by_id" btree (created_by_id)
    "index_activity_streams_on_group_id" btree (group_id)
    "index_activity_streams_on_id" btree (id) WITH (fillfactor='90')
    "index_activity_streams_on_o_id" btree (o_id)
    "index_activity_streams_on_permission_id" btree (permission_id)
    "index_activity_streams_on_permission_id_and_group_id" btree (permission_id, group_id)
    "index_activity_streams_on_permission_id_group_id_created_at" btree (permission_id, group_id, created_at)
Foreign-key constraints:
    "fk_rails_15ed0d0859" FOREIGN KEY (activity_stream_type_id) REFERENCES type_lookups(id)
    "fk_rails_2abed7f6ca" FOREIGN KEY (permission_id) REFERENCES permissions(id)
    "fk_rails_5e0981116b" FOREIGN KEY (created_by_id) REFERENCES users(id)
    "fk_rails_9006c69204" FOREIGN KEY (group_id) REFERENCES groups(id)
    "fk_rails_add7ae94d9" FOREIGN KEY (activity_stream_object_id) REFERENCES object_lookups(id)
Has OIDs: no

So to continue I wanted to update created_by_id to 1 to be able to continue. This might be incorrect from workflow point of view, but I wanted to see what happens.

However, I couldn't find any ActivitySteams by affected user IDs.
I also checked for o_id to double tab, same result.

2.6.6 :001 > ActivityStream.where(created_by_id: 235017).count
 => 0
2.6.6 :002 > ActivityStream.where(created_by_id: 256036).count
 => 0

So what I did then was, I dropped all 437,783 ActivityStreams.
What ever happens with the ActivityStreams... it's no longer there after rolling back. Possibly we really create at least one entry in the stream

If you provide debugging code I'll implement it the next time this happens.

If I run User.find(256036).destroy manually, the error persists

@rolfschmidt
Copy link
Collaborator

checkout this place:

zammad/app/models/activity_stream.rb

Line 32 in c3b1b44

def self.add(data)

put the following the beginning of the function:

Rails.logger.error "ActivityStream.add - #{data.inspect}"

Maybe it will run when you are trying to destroy the user.

@rolfschmidt
Copy link
Collaborator

It would be also interesting if this change fixes the problem:

zammad/app/models/user.rb

Line 1172 in c3b1b44

user_columns = %w[created_by_id updated_by_id origin_by_id owner_id archived_by_id published_by_id internal_by_id]

Replace:

    user_columns = %w[created_by_id updated_by_id origin_by_id owner_id archived_by_id published_by_id internal_by_id]

with

    user_columns = %w[updated_by_id created_by_id origin_by_id owner_id archived_by_id published_by_id internal_by_id]

@MrGeneration
Copy link
Member Author

Maybe it will run when you are trying to destroy the user.

See my comment above, this fails as well.

So initially I only updated activity_stream.rb and can indeed tell that it does create ActivityStreams....
I did filter for the user ID only, here's the relevant log lines:

I, [2021-02-12T13:52:04.761263 #24634-12253600]  INFO -- : Enqueued SearchIndexJob (Job ID: c366e837-ef18-408f-b01c-81fa85048f1d) to DelayedJob(default) with arguments: "User", 235017
I, [2021-02-12T13:52:04.766917 #24634-12253600]  INFO -- : Won't enqueue SearchIndexJob (Job ID: a57fca87-0981-4f31-b8b2-e45fd8bc473b) because of already existing job with lock key 'SearchIndexJob/User/235017'.
I, [2021-02-12T13:52:04.767042 #24634-12253600]  INFO -- : Enqueued SearchIndexJob (Job ID: a57fca87-0981-4f31-b8b2-e45fd8bc473b) to DelayedJob(default) with arguments: "User", 235017
I, [2021-02-12T13:52:04.772151 #24634-12253600]  INFO -- : Won't enqueue SearchIndexJob (Job ID: 8bbaa863-73e5-4e1b-9723-8786f7926d0f) because of already existing job with lock key 'SearchIndexJob/User/235017'.
I, [2021-02-12T13:52:04.772251 #24634-12253600]  INFO -- : Enqueued SearchIndexJob (Job ID: 8bbaa863-73e5-4e1b-9723-8786f7926d0f) to DelayedJob(default) with arguments: "User", 235017
I, [2021-02-12T13:52:04.777065 #24634-12253600]  INFO -- : Won't enqueue SearchIndexJob (Job ID: 09675985-b427-4dd7-b705-6e0d3f570e48) because of already existing job with lock key 'SearchIndexJob/User/235017'.
I, [2021-02-12T13:52:04.777179 #24634-12253600]  INFO -- : Enqueued SearchIndexJob (Job ID: 09675985-b427-4dd7-b705-6e0d3f570e48) to DelayedJob(default) with arguments: "User", 235017
I, [2021-02-12T13:52:04.782556 #24634-12253600]  INFO -- : Won't enqueue SearchIndexJob (Job ID: 94843eea-4c8b-49ea-9ee1-b2ec887e3e86) because of already existing job with lock key 'SearchIndexJob/User/235017'.
I, [2021-02-12T13:52:04.782672 #24634-12253600]  INFO -- : Enqueued SearchIndexJob (Job ID: 94843eea-4c8b-49ea-9ee1-b2ec887e3e86) to DelayedJob(default) with arguments: "User", 235017
I, [2021-02-12T13:52:06.290291 #22773-80092120]  INFO -- : Started GET "/api/v1/users/235017?full=true&_=1613117540190" for 95.91.247.183 at 2021-02-12 13:52:06 +0100
I, [2021-02-12T13:52:06.293546 #22773-80092120]  INFO -- :   Parameters: {"full"=>"true", "_"=>"1613117540190", "id"=>"235017"}
E, [2021-02-12T13:52:06.555071 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>373270, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.571492 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>373287, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.583591 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374069, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.594086 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374103, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.606140 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374123, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.619450 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374132, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.630586 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374154, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.641077 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374317, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.653280 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374353, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.663925 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374368, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.673922 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374375, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.684646 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374423, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.696720 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374612, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.706860 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>374642, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.718882 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>376170, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.730057 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>376222, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.738738 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>376279, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.746669 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>376293, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.765583 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>376896, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.775648 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>376918, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.783650 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377090, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.790673 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377121, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.797881 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377172, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.805583 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377195, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.812735 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377643, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.820101 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377658, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.828089 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377684, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.837154 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377722, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.845596 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>377739, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.855615 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>378520, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.864245 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>378535, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.872182 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>378539, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.881454 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>378563, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.889824 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>381402, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.897208 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>381529, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.904864 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>382807, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.920371 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>383644, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.929059 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>384555, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.937710 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>384592, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.946267 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>384707, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.954190 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>386756, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.963701 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>386839, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.972290 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>386925, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.985517 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>387019, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:06.994036 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>392894, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:06 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.004219 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>394308, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.015698 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>395655, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.027174 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411773, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.035401 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411775, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.043307 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411810, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.050740 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411821, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.058442 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411830, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.068894 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411915, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.083182 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411931, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.092129 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411974, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.102575 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411978, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.120466 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>411992, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.127984 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412014, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.134930 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412164, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.142121 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412204, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.151744 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412207, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.160933 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412234, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.169160 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412257, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.175998 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412268, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.182679 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412299, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.189430 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412354, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.196761 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412366, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.203885 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412397, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.210616 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412409, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.217303 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412416, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.224525 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412434, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.232780 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412450, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.241555 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412453, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.249908 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412463, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.258301 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412483, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.278665 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412485, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.287656 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412520, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.296314 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412523, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.308812 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412535, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.318490 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412944, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.327143 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>412963, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.335781 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413025, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.344524 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413112, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.352052 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413156, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.359369 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413168, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.367090 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413189, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.376616 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413204, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.388120 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413325, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.398880 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413336, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.407571 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413352, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.415461 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413380, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.423144 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413399, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.430372 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413771, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.438598 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413783, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.447022 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413802, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.467833 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413809, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.475595 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413819, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.488645 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413909, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.498488 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413930, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.506368 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413937, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.514699 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>413958, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.526487 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>414358, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.537684 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>414360, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.547479 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>414510, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.558283 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>414640, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.569061 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>414656, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.578071 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>414812, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.588531 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>414996, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.602610 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>415010, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.612422 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>415014, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.620373 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>415065, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.629373 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>415098, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.637063 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>415127, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.644391 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>415174, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.662440 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>416569, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.671318 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>416579, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.680499 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>416665, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.689216 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>416746, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.697205 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417143, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.705178 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417218, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.713723 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417222, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.724974 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417279, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.737525 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417297, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.745336 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417339, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.753424 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417347, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.762561 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417866, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.770879 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417911, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.779294 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417920, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.789358 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417955, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.800695 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417965, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.811367 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>417980, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.819222 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>418000, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.827104 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>418247, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.844752 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>418256, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.853090 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>418273, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.861131 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>418744, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.869577 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>418784, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.879709 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419321, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.889750 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419371, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.900142 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419379, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.910891 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419386, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.920019 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419486, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.927488 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419504, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.934832 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419509, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.942341 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419521, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.950082 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419526, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.959251 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419558, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.967061 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419564, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.974849 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419567, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.983106 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>419766, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.991154 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>420267, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:07.998826 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>420358, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:07 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.018542 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>420470, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.026600 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>420500, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.034049 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>426550, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.046138 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>426725, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.059153 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>427487, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.071495 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>427819, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.080872 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>427832, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.089341 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>427865, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.102434 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>429863, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.117843 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>430319, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.141756 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>430337, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.162279 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>430351, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.180782 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>431334, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.204687 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483089, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.225739 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483099, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.241305 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483107, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.257225 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483113, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.276783 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483118, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.299481 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483130, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.326720 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483139, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.349520 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483147, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.361505 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483732, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.373864 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483738, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.388000 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483744, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.398563 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483748, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.411611 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483750, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
E, [2021-02-12T13:52:08.422297 #24634-12253600] ERROR -- : ActivityStream.add - {:o_id=>483765, :type=>"update", :object=>"Ticket::Article", :group_id=>2, :permission=>"ticket.agent", :created_at=>Fri, 12 Feb 2021 12:52:08 UTC +00:00, :created_by_id=>235017}
DETAIL:  Key (id)=(235017) is still referenced from table "activity_streams".

I then moved over to adjusting user.rb - just a side note, on the stable 3.6 instance of the customer the affected line is 2 lines below the mentioned here. I did change the line anyway.

Changing the order does not fix the issue, basically the log out put from above looks kinda the same tbh.

@rolfschmidt
Copy link
Collaborator

oh nice debug infos!! Nice thank you very much... i think i know now where the problem is let me take a look...

zammad-sync pushed a commit that referenced this issue Mar 2, 2021
@rolfschmidt @thorsteneckel
Fixes #3402 - DataPrivacy Task fails in some situations.
b9623e3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug data privacy prioritised by payment verified
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mantas @rolfschmidt @MrGeneration