Hi All,
Unable to route the json logs based on a a keyword (regex ) "MyService_DataApp" on the event to a particular index testlogs_idx .Could you please point anything wrong with the below and these configurations are on Heavy forwarder ,SH's and Indexers.
To test the routing I've created an index=thisshouldneverhappen and added under the inputs , and set up an alert, whenever an event hits that index to know something is broken , all the events still route to the index=thisshouldneverhappen .
Props
[json_srctype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+){
NO_BINARY_CHECK=true
KV_MODE=json
MAX_TIMESTAMP_LOOKAHEAD=45
TIME_PREFIX=\W+\w{8}
TIME_FORMAT=%s%3N
TRUNCATE=50000
ANNOTATE_PUNCT=false
disabled=false
pulldown_type =true
TRANSFORMS-01_testlogs= a1-testlogs-Route
TRANSFORMS-02_testlogs =a2-testlogs-SourceType
Transforms
[a1-testlogs-Route]
DEST_KEY = _MetaData:Index
REGEX = MyService_DataApp
FORMAT = testlogs_idx
[a2-testlogs-SourceType]
DEST_KEY = MetaData:Sourcetype
REGEX = MyService_DataApp
FORMAT = sourcetype::testlogs_srctype
the routing to index and sourcetpe are working after making changes to the json logs where the keyword (regex ) "MyService_DataApp" is at the last of the event under the nested xml and I've moved the keyword (regex ) "MyService_DataApp" to the top under metadata with json key value pair
All the configs are correct .
the routing to index and sourcetpe are working after making changes to the json logs where the keyword (regex ) "MyService_DataApp" is at the last of the event under the nested xml and I've moved the keyword (regex ) "MyService_DataApp" to the top under metadata with json key value pair
All the configs are correct .
Hi mahesh423,
to debug your situation, try the following checks (probably someone of them you already used!):
and let me know the new situation.
Bye.
Giuseppe
Thanks @gcusello.I'm giving a few points which I tried . please review and advice.
Steps 1 completed - I've props and transforms on the Heavy forwarder and search heads and indexers and restarted .
2 - Steps completed .validated using https://regexr.com/
Step 3 -
try to insert in props.conf also a stanza for the new sourcetype "testlogs_srctype" - Not sure how can I add the new sourcetype as per the props -[json_srctype] , TRANSFORMS-02_testlogs =a2-testlogs-SourceType
attribute needs to create a new sourcetype , same logic works for with other data.
[a2-testlogs-SourceType]
DEST_KEY = MetaData:Sourcetype
REGEX = MyService_DataApp
FORMAT = sourcetype::testlogs_srctype
The above same logic is working with xml format data . the new data has
using the keyword "MyService_DataApp" within a json format xml nested object - embedded with in the as below
"LogMessage": {
"request": "MyService_DataApp"
transforms.conf , regex used as below
REGEX = MyService_DataApp
my regex pattern in the json log is at 6089 column and ran btool on the heavy forwarder for props and validated the DEPTH LIMIT which is default 1000 and what could be the reason or needs increase ( chances of bad performance) and MATCH_LIMIT is default which is 100000.