RestResource to log results of integrations on parent transaction

[dschibster]

Hi @jongpie,

happy to report that since I presented Nebula Logger at the Salesforce Architects meetup in Düsseldorf, there's hardly a project where I don't see it in use. :)

Right now in one of our projects, we are also using it to log all of our Apex Transactions. However, and this is where things become interesting, we are integrating with MuleSoft. MuleSoft is asynchronously handling the transformation and load into different systems (and Salesforce too). All we get back at the time of sending our payload is a Success.

Of course, we also want to get to know about errors happening in MuleSoft. We were contemplating creating a RestResource for external Systems to give information back to us, for example by creating a new Log (As Parent Log) under the one that triggered the integration itself.

Before we go into further detail creating this ourselves (because architectural guidelines are hestitant about letting us create endpoints), what do you think, would you consider it a worthwile addition to Nebula Logger in general?

BR
Dennis

[jongpie]

Hi @dschibster that's incredible to hear that you're seeing so much usage of Nebula Logger! 🥳

I think the idea of adding a RestResource endpoint is definitely worth considering - I've built & used a few basic prototypes in the past for this, but I never developed the prototypes enough to actually release them here in GitHub. I think that there are 2 valid use cases for having a RestResource class for Nebula Logger:

1. Monitoring of external systems. Your scenario is a perfect example - an external system (Mulesoft, but could be any external system) is doing some off-platform work, but it's relevant to work being done on-platform, so having a way to create & associate logs would provide extra monitoring to Salesforce admins/developers of external systems.
2. Data portability. Right now, Nebula Logger's data is always created - and stays - in a single Salesforce org. But I think there is possible some value in being able to easily move some of Nebula Logger's data to other systems - including from one Salesforce org to another.
   • There have been a few discussions, including this one, about using a standard like OpenTelemetry to send data from a Salesforce org (outbound) to another system, such as an external log analysis tool (like Splunk, DataDog, Beehive, etc.). This wouldn't directly require a RestResource itself, but this is would handle the "outbound" part of improving Nebula Logger's data portability
   • By also adding a RestResource to Nebula Logger (inbound), you could potentially have 1 Salesforce org send its Nebula Logger data to another Salesforce org. This would provide larger/more complex teams with the ability to centralize logging data from multiple Salesforce orgs into a single org (e.g., all logs from a team's scratch orgs and developer sandboxes could have their logs automatically sent to a full-copy sandbox for storage)

I completely agree with being hesitant about introducing new REST endpoints, and I don't think this feature would be needed by everyone using Nebula Logger. But for orgs with a lot of external integrations, multiple environments, and other similar complexities, I think having a RestResource endpoint built into Nebula Logger could be very useful. Depending on how its built, it could be implemented in a way that helps with both monitoring of external systems and data portability.

Let me know what you think!

[jongpie]

Hey @dschibster I would love to review a PR, if you're interested in submitting one! I think I would want to initially release this as a separate unlocked package (that depends on Nebula Logger's core unlocked package, similar to the other existing plugins), and then later move the RestResource into the core package (once the functionality has been refined & enhanced). This would help me in a couple of ways:

• Since RestResource classes use global, there are some limitations on what can change in future releases of the managed package. Releasing it initially as a separate package will make it easier to experiment & expand on the functionality, without needing to worry about the impact to the managed package.
• Long-term, I do think that using OpenTelemetry's modal should be supported by the RestResource - so I'd like to eventually enhance the RestResource to support that type of data model. But this doesn't have to be there in your PR, as it wouldn't benefit your use case anyway, and releasing it in a separate package gives more control on what changes we make in the future.

I've been using plugins as a way to test out new, experimental features - my plan long-term is to eventually merge most of the existing plugins into the core package. So far, it's been a great way to provide extra functionality to orgs that use Nebula Logger, while providing myself some extra flexibility in the development process - and I think this is another great candidate to follow that approach.

With that said, if you're ok with that plan of making this a plugin initially, then I think you've got the right idea - the RestResource would need to have some type of parameters object/options, and then you'd call setScenario, setParentTransactionId, setTags, etc. And I completely agree, you would want to always call saveLog() at the end of the RestResource's method.

I've created issue #635 to officially track this as an enhancement. Let me know if you have follow-up questions on how you'd like to implement the RestResource. And feel free to submit a PR whenever you have something to review - even if it's not finished, I'm happy to take a look & provide feedback early on (if that would be helpful to you).

[jongpie]

Hi @dschibster - just wanted to follow up with you on this, are you still considering contributing a PR for some of this? If not, that's no problem, I'm happy to work on it instead, just let me know what works for you.

[dschibster]

Hi @jongpie, sitting in Japan for vacation currently so best wishes! I am still planning to do it, however I can't give you a concrete timeline due to timeline constraints on the client side.

My preference would be to just create logs in the Salesforce instance without transformation for external logging systems, so if you want to tackle that, I wouldn't interfere with you. Or you'd build on top of it.

Are you planning to work on this in the coming weeks?

[jongpie]

@dschibster that sounds incredible, hope you're enjoying your vacation in Japan!

And that sounds good, I'll plan to look into setting up some transformations (using OpenTelemetry's format). I might start sometime in the coming weeks, but I think it'll mostly be exploratory work for now - it might be a few months before I have something fully built out.

[dschibster]

Sounds good! I'll keep you posted. We got an MVP rollout in May so probably thereafter I can shift focus.

[anengineerdude]

Came across this discussion for a very similar use case. I am implementing several integrations to SF via Mulesoft and heavily debating where Mulesoft logging should lie. While there is still merit in something like OpenTelemetry for mass ingestion of Mule logs, having some consise logging for scheduled integrations and simplistic error handling at a high level that would should up in NebulaLogger would be a great UX for the SF Admins.

A RestResource certainly seems like a great way to doing this. Looking forward to it and happy to test out any ideas. While I am getting good with SF admin and Mule, my package development and Apex skill is still pretty basic, so I am not to much help on the development side.

[dschibster]

Thanks for the feedback! I still have this on my radar, the release timeline that I have to focus on currently though has been a little delayed. When things get softer, I should be able to handle this and get a PR running for @jongpie to review.