Write middleware to track user activity #150

kmehant · 2020-07-14T16:47:44Z

Signed-off-by: K mehant 411843@student.nitandhra.ac.in

Context

This pull request adds an express middleware that can be used to track user activity and store them in a Mongo database.

Design

Taking several points from @vaibhavdaren and @Rupeshiya, I have come with this approach in designing the middleware

All requests other than get are recorded as below data models into Redis cache
When logout was done, complete Redis cache is parsed and pushed into the MongoDB User model -> activity []
Redis cache is then cleared.

Data model on Redis:
A Redis List for cache

userID => [ "route,collection,method,objectID", "route,collection,method,objectID",  ...]

Data model on Mongo:

User: 
.....
.....
 activity: [{
    route: {
      type: String,
      required: true
    },
    method: {
      type: String,
      required: true
    },
    collectionType: {
      type: String,
      required: true
    },
    id: {
      type: String,
      required: true
    },
  }]

method request method
route name of the route
id Mongodb`s _id or object id (taken from the response)
collection type : Type of the collection where we can use our id to fetch data such as created at and others.

partially closes #148

Rupeshiya

@kmehant We should not only record the timestamp and routes but what we want is basically the activity of the user let say a user "X" is creating a post so what he will do:

Send POST request to /post
With some data
So it should store the activity like: userId, postId, timeStamp, so that we when we render in the client side then from there on clicking a particular activity an admin can navigate to that post.

Also, we should not call the middleware on GET requests as we are not interested in what the user is exploring on the platform but we are interested in what actions he/she is performing on the platform.

kmehant · 2020-07-15T11:40:12Z

@kmehant We should not only record the timestamp and routes but what we want is basically the activity of the user let say a user "X" is creating a post so what he will do:

Send POST request to /post

With some data
So it should store the activity like: userId, postId, timeStamp, so that we when we render in the client side then from there on clicking a particular activity an admin can navigate to that post.

Also, we should not call the middleware on GET requests as we are not interested in what the user is exploring on the platform but we are interested in what actions he/she is performing on the platform.

@Rupeshiya
Thank you for the review
I had this doubt, I guess I spoke about the data model in the issue for your approval 🥺☹️ before implementing it.
For get requests thing, I haven't used the middleware yet, we can include this middleware to any route we wish to use.

@Rupeshiya @vaibhavdaren @devesh-verma
Please 🙏🙏 I guess it would be easy if we can discuss on the design, approach correctly and the outcomes so that we all will be on the same page ✌️ and intention.

Rupeshiya · 2020-07-15T15:03:59Z

app/middleware/activity.js

+    var routeNamesDict
+    var mailid
+    const data = await User.findOne({
+      'tokens.token': token


use user-id instead of the token as by default indexing is done on _id in mongoose so it would give better performance.

vaibhavdaren · 2020-07-15T16:32:33Z

HASH   ---> (KEY, HASH_VALUE)
						|				 
(USER)       |         (RANDOM)
		   (ROUTE)
		      |
		IS THIS UNIQUE?

			KEY           HASH		 ID_OF_OBJECT_RETURNED
 HASH  ---> TIMESTAMP ---> ROUTE -->      (UUID_mogo_id)
    |
(per_user)

Rupeshiya

@kmehant Awesome!!
Also I think we need to add this middleware in every routes except GET requests. In addition to that do you think we need to track user login and logout??

Also will it work in case of comment, because in that case we want postId as well as commentId??

Rupeshiya · 2020-07-16T13:00:22Z

app/middleware/activity.js

+      activityElement.method = activityDataElement[1]
+      activityElement.collectionType = activityDataElement[2]
+      activityElement.id = activityDataElement[3]
+      data.activity.push(activityElement)


change push to unshift, we want recent activity as first element.

Rupeshiya · 2020-07-18T06:27:37Z

@kmehant Also please add the API to fetch those user activity data from DB.

kmehant · 2020-07-18T07:22:39Z

Also will it work in case of comment, because in that case we want postId as well as commentId??

@Rupeshiya Yes, it should work!
based on the response as I can see postid and commentid keys in the response object already. So we can use them.

kmehant · 2020-07-18T07:27:11Z

app/middleware/activity.js

+    console.log(data)
+    // clear data from redis
+    await redisClient.del(userID)
+  } else if (method != 'GET') {


@kmehant Awesome!!
Also I think we need to add this middleware in every routes except GET requests. In addition to that do you think we need to track user login and logout??

@Rupeshiya I have written check condition here so we can add it everywhere safely 😁

kmehant

@Rupeshiya

kmehant · 2020-07-18T07:36:43Z

@Rupeshiya Can you suggest API endpoints so that I can move forward

Rupeshiya · 2020-07-18T15:10:03Z

@Rupeshiya Can you suggest API endpoints so that I can move forward

Yeah, let it be /user/:id/activity (here id is the userId as we want to fetch for each user by their id)

Signed-off-by: K mehant <411843@student.nitandhra.ac.in>

vaibhavdaren · 2020-08-13T14:27:11Z

@kmehant ETA?

kmehant changed the title ~~[WIP] Activity tracking for admin~~ [WIP] Write middleware to track user activity Jul 15, 2020

kmehant force-pushed the activity-patch branch from 096ac6d to 0cd9dd3 Compare July 15, 2020 08:07

kmehant mentioned this pull request Jul 15, 2020

Implement API to showcase activities of a user to the admins #148

Open

vaibhavdaren requested review from devesh-verma and Rupeshiya July 15, 2020 08:20

kmehant changed the title ~~[WIP] Write middleware to track user activity~~ Write middleware to track user activity Jul 15, 2020

Rupeshiya requested changes Jul 15, 2020

View reviewed changes

Rupeshiya reviewed Jul 15, 2020

View reviewed changes

kmehant requested a review from Rupeshiya July 16, 2020 08:53

Rupeshiya reviewed Jul 16, 2020

View reviewed changes

kmehant commented Jul 18, 2020

View reviewed changes

kmehant added 7 commits July 23, 2020 20:55

Fix conflicts

b56f979

Signed-off-by: K mehant <411843@student.nitandhra.ac.in>

Resolve few bugs and some progress

a80bd72

Signed-off-by: K mehant <411843@student.nitandhra.ac.in>

Fix mongo connection bugs

9318b4c

Signed-off-by: K mehant <411843@student.nitandhra.ac.in>

Delete data from redis when logout

a0c2910

Signed-off-by: K mehant <411843@student.nitandhra.ac.in>

Update middleware

0f850ae

Signed-off-by: K mehant <411843@student.nitandhra.ac.in>

Update User model

e40df1e

Signed-off-by: K mehant <411843@student.nitandhra.ac.in>

Add middleware to routers and also add API for activity

48b32e6

Signed-off-by: K mehant <411843@student.nitandhra.ac.in>

kmehant force-pushed the activity-patch branch from e352270 to 48b32e6 Compare July 23, 2020 15:28

This was referenced Aug 15, 2020

hot fix Rupeshiya/social-platform-donut-backend#1

Merged

User activity tracker using redis #174

Merged

vaibhavdaren added the abandoned label Oct 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write middleware to track user activity #150

Write middleware to track user activity #150

kmehant commented Jul 14, 2020 •

edited

Rupeshiya left a comment

kmehant commented Jul 15, 2020

Rupeshiya Jul 15, 2020

vaibhavdaren commented Jul 15, 2020 •

edited

Rupeshiya left a comment •

edited

Rupeshiya Jul 16, 2020

Rupeshiya commented Jul 18, 2020

kmehant commented Jul 18, 2020

kmehant Jul 18, 2020

kmehant left a comment

kmehant commented Jul 18, 2020

Rupeshiya commented Jul 18, 2020 •

edited

vaibhavdaren commented Aug 13, 2020

Write middleware to track user activity #150

Are you sure you want to change the base?

Write middleware to track user activity #150

Conversation

kmehant commented Jul 14, 2020 • edited

Context

Design

Rupeshiya left a comment

Choose a reason for hiding this comment

kmehant commented Jul 15, 2020

Rupeshiya Jul 15, 2020

Choose a reason for hiding this comment

vaibhavdaren commented Jul 15, 2020 • edited

Rupeshiya left a comment • edited

Choose a reason for hiding this comment

Rupeshiya Jul 16, 2020

Choose a reason for hiding this comment

Rupeshiya commented Jul 18, 2020

kmehant commented Jul 18, 2020

kmehant Jul 18, 2020

Choose a reason for hiding this comment

kmehant left a comment

Choose a reason for hiding this comment

kmehant commented Jul 18, 2020

Rupeshiya commented Jul 18, 2020 • edited

vaibhavdaren commented Aug 13, 2020

kmehant commented Jul 14, 2020 •

edited

vaibhavdaren commented Jul 15, 2020 •

edited

Rupeshiya left a comment •

edited

Rupeshiya commented Jul 18, 2020 •

edited