Splunk Enterprise

Best Logging Method with Splunk

bahadirbal
New Member

We are using micro Services in our system. All services are doing its tast and call other one. We are logging these services requests/responses with a same uniqueIdentifier. What do you suggest to see this flow from point to endpoint in one query with inputs and outputs in same row according to timestamps? We need to monitor our system from begging to end.

Or maybe you can suggest a method/way us to log differently to achieve this goal, we are open for any suggestion.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Relying on timestamp in a distributed system may not be good enough, because you will probably never be able to guarantee exact timestamp synchronization. A better approach to show flow later in search is to log parent/child uniqueId:

Seq      Parent       Current
1         null          id1
2         id1           id2
3         id2           id3
4         id2           id4
etc. 

This will allow you to not only show a call sequence, but also parallel execution paths (see Seq 3&4 above), if your system allows for those to happen.
But having a uniqueId in all log messages will be enough to show everything that happened in context, just potentially not in exact sequence if there is clock drift between systems.

jplumsdaine22
Influencer

Read this: http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6

And yes, if you want to be able to easily identify transactions later, then include the transaction identifier in each log entry - that way you're not at the mercy of incorrectly configured ntp services (although definately include the timestamp ! As precise as your system allows)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...