Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

script to discover the dead pubsub subscriptions #1857

Open
1DevOps2 opened this issue Nov 27, 2023 · 2 comments
Open

script to discover the dead pubsub subscriptions #1857

1DevOps2 opened this issue Nov 27, 2023 · 2 comments
Labels
api: pubsub Issues related to the googleapis/nodejs-pubsub API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: question Request for information or clarification. Not an issue.

Comments

@1DevOps2
Copy link

Hi there, I am currently facing an issue while attempting to discover dead subscriptions in Google Cloud Pub/Sub using Node.js scripts.

Background:
I recently set up an alert system on a Pub/Sub topic to send a notification to Slack if the backlog ever exceeds 50 messages. However, I encountered a situation where the message backlog reached over 200K messages. To address this, I am now trying to create a script that can identify and clean up dead Pub/Sub subscriptions.

Issue:
While I am not very familiar with Node.js, I attempted to create a scripts using the @google-cloud/pubsub library. Unfortunately, neither of these scripts seems to be working as expected.

script:

const { PubSub } = require('@google-cloud/pubsub'); 
  
// Replace 'your-project-id' with your actual Google Cloud Project ID 
const projectId = 'your-project-id'; 
  
// Creates a PubSub client 
const pubsub = new PubSub({ projectId }); 
  
async function discoverDeadSubscriptions() { 
 try { 
   // List all subscriptions in the project 
   const [subscriptions] = await pubsub.getSubscriptions(); 
  
   // Check each subscription for activity 
   for (const subscription of subscriptions) { 
     const [messages] = await pubsub 
       .topic(subscription.metadata.topic) 
       .subscription(subscription.name) 
       .seekToTime('2023-01-01T00:00:00Z') // Seek to the beginning of time 
       .get(); 
  
     if (messages.length === 0) { 
       console.log(`Subscription ${subscription.name} is empty (no messages). It may be a candidate for cleanup.`); 
       // You can add code here to perform cleanup actions for dead subscriptions 
     } else { 
       console.log(`Subscription ${subscription.name} has messages and is active.`); 
     } 
   } 
 } catch (error) { 
    console.error(`Error discovering dead subscriptions: ${error.message}`); 
 } 
} 
  
// Run the function to discover dead subscriptions 
discoverDeadSubscriptions();

another:

const { PubSub } = require('@google-cloud/pubsub'); 
const pubsub = new PubSub(); 
const projectName = 'YOUR_PROJECT_ID'; 

const checkDeadSubscriptions = async () => { 
  const subscriptions = await pubsub.getSubscriptions(); 

  for (const subscription of subscriptions) { 
    const subscriptionName = subscription.name; 
    const subscriptionData = await pubsub.subscriptionMetadata(subscriptionName); 
    const lastAcknowledged = subscriptionData.ackId; 
    const lastPublished = subscriptionData.pubsubMessage.publishTime; 
    const timeSinceLastMessage = Date.now() - lastPublished; 

    // Check if it has been more than one month since the last message was published 
    if (timeSinceLastMessage > 2592000000) { // 1 month in milliseconds 
      console.log(`Potentially dead subscription: ${subscriptionName}`); 
    } 
  } 
}; 
checkDeadSubscriptions(); 

I'd greatly appreciate your help in resolving the issues with these scripts.
Thanks a bunch for your time and assistance.

Best regards,

@1DevOps2 1DevOps2 added priority: p3 Desirable enhancement or fix. May not be included in next release. type: question Request for information or clarification. Not an issue. labels Nov 27, 2023
@product-auto-label product-auto-label bot added the api: pubsub Issues related to the googleapis/nodejs-pubsub API. label Nov 27, 2023
@feywind
Copy link
Collaborator

feywind commented Dec 18, 2023

I can't really help with code writing but I do think it's worth asking, why do those subscriptions go dark? Is it something you expect in your app, or do messages just quit being delivered, or what?

@1DevOps2
Copy link
Author

Hello, thank you for your response. I would like to delete a message backlog that has accumulated to over 200K messages. To achieve this, I need to identify and remove old, inactive subscriptions among the 130 pubsub subscriptions I have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsub Issues related to the googleapis/nodejs-pubsub API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: question Request for information or clarification. Not an issue.
Projects
None yet
Development

No branches or pull requests

2 participants