All Apps and Add-ons

Split twitter events in multiple events

celianouguier
Explorer

Hi everyone,

I get tweets via the twitter API on Splunk. This allows me to retrieve several tweets in the same event. In the json, there is a statuses array that contains several tweets.

I would like to have only one line per tweet, either through a command in a search or directly when retrieving data. But I have no idea how to do it, can you help me?

At the end of the json, or the statuses table, we have the "search_metadata". If it helps, we can lose that information, I don't mind.

This is the format of the json:

{
        "statuses": [
                {
                        "created_at": "Sun Feb 25 18:11:01 +0000 2018",
                        "id": 967824267948773377,
                        "id_str": "967824267948773377",
                        "text": "From pilot to astronaut, Robert H. Lawrence was the first African-American to be selected as an astronaut by any na… https://t.co/FjPEWnh804",
                        "truncated": true,
                        "entities": {
                                "hashtags": [],
                                "symbols": [],
                                "user_mentions": [],
                                "urls": [
                                        {
                                                "url": "https://t.co/FjPEWnh804",
                                                "expanded_url": "https://twitter.com/i/web/status/967824267948773377",
                                                "display_url": "twitter.com/i/web/status/9…",
                                                "indices": [
                                                        117,
                                                        140
                                                ]
                                        }
                                ]
                        },
                        "metadata": {
                                "result_type": "popular",
                                "iso_language_code": "en"
                        },
                        "source": "<a href="https://www.sprinklr.com" rel="nofollow">Sprinklr</a>",
                        "in_reply_to_status_id": null,
                        "in_reply_to_status_id_str": null,
                        "in_reply_to_user_id": null,
                        "in_reply_to_user_id_str": null,
                        "in_reply_to_screen_name": null,
                        "user": {
                                "id": 11348282,
                                "id_str": "11348282",
                                "name": "NASA",
                                "screen_name": "NASA",
                                "location": "",
                                "description": "Explore the universe and discover our home planet with @NASA. We usually post in EST (UTC-5)",
                                "url": "https://t.co/TcEE6NS8nD",
                                "entities": {
                                        "url": {
                                                "urls": [
                                                        {
                                                                "url": "https://t.co/TcEE6NS8nD",
                                                                "expanded_url": "http://www.nasa.gov",
                                                                "display_url": "nasa.gov",
                                                                "indices": [
                                                                        0,
                                                                        23
                                                                ]
                                                        }
                                                ]
                                        },
                                        "description": {
                                                "urls": []
                                        }
                                },
                                "protected": false,
                                "followers_count": 28605561,
                                "friends_count": 270,
                                "listed_count": 90405,
                                "created_at": "Wed Dec 19 20:20:32 +0000 2007",
                                "favourites_count": 2960,
                                "utc_offset": -18000,
                                "time_zone": "Eastern Time (US & Canada)",
                                "geo_enabled": false,
                                "verified": true,
                                "statuses_count": 50713,
                                "lang": "en",
                                "contributors_enabled": false,
                                "is_translator": false,
                                "is_translation_enabled": false,
                                "profile_background_color": "000000",
                                "profile_background_image_url": "http://pbs.twimg.com/profile_background_images/590922434682880000/3byPYvqe.jpg",
                                "profile_background_image_url_https": "https://pbs.twimg.com/profile_background_images/590922434682880000/3byPYvqe.jpg",
                                "profile_background_tile": false,
                                "profile_image_url": "http://pbs.twimg.com/profile_images/188302352/nasalogo_twitter_normal.jpg",
                                "profile_image_url_https": "https://pbs.twimg.com/profile_images/188302352/nasalogo_twitter_normal.jpg",
                                "profile_banner_url": "https://pbs.twimg.com/profile_banners/11348282/1518798395",
                                "profile_link_color": "205BA7",
                                "profile_sidebar_border_color": "000000",
                                "profile_sidebar_fill_color": "F3F2F2",
                                "profile_text_color": "000000",
                                "profile_use_background_image": true,
                                "has_extended_profile": true,
                                "default_profile": false,
                                "default_profile_image": false,
                                "following": null,
                                "follow_request_sent": null,
                                "notifications": null,
                                "translator_type": "regular"
                        },
                        "geo": null,
                        "coordinates": null,
                        "place": null,
                        "contributors": null,
                        "is_quote_status": false,
                        "retweet_count": 988,
                        "favorite_count": 3875,
                        "favorited": false,
                        "retweeted": false,
                        "possibly_sensitive": false,
                        "lang": "en"
                },
                {
                        "created_at": "Mon Feb 26 01:07:05 +0000 2018",
                        "id": 967928974960545793,
                        "id_str": "967928974960545793",
                        "text": "Congratulations to #Olympics athletes who won gold! Neutron stars like the one at the heart of the Crab Nebula may… https://t.co/vz4SnPupe2",
                        "truncated": true,
                        "entities": {
                                "hashtags": [
                                        {
                                                "text": "Olympics",
                                                "indices": [
                                                        19,
                                                        28
                                                ]
                                        }
                                ],
                                "symbols": [],
                                "user_mentions": [],
                                "urls": [
                                        {
                                                "url": "https://t.co/vz4SnPupe2",
                                                "expanded_url": "https://twitter.com/i/web/status/967928974960545793",
                                                "display_url": "twitter.com/i/web/status/9…",
                                                "indices": [
                                                        116,
                                                        139
                                                ]
                                        }
                                ]
                        },
                        "metadata": {
                                "result_type": "popular",
                                "iso_language_code": "en"
                        },
                        "source": "<a href="https://studio.twitter.com" rel="nofollow">Media Studio</a>",
                        "in_reply_to_status_id": null,
                        "in_reply_to_status_id_str": null,
                        "in_reply_to_user_id": null,
                        "in_reply_to_user_id_str": null,
                        "in_reply_to_screen_name": null,
                        "user": {
                                "id": 19802879,
                                "id_str": "19802879",
                                "name": "NASA JPL",
                                "screen_name": "NASAJPL",
                                "location": "Pasadena, Calif.",
                                "description": "NASA Jet Propulsion Laboratory manages many of NASA's robotic missions exploring Earth, the solar system and our universe. Tweets from JPL's News Office.",
                                "url": "http://t.co/gcM9d1YLUB",
                                "entities": {
                                        "url": {
                                                "urls": [
                                                        {
                                                                "url": "http://t.co/gcM9d1YLUB",
                                                                "expanded_url": "http://www.jpl.nasa.gov",
                                                                "display_url": "jpl.nasa.gov",
                                                                "indices": [
                                                                        0,
                                                                        22
                                                                ]
                                                        }
                                                ]
                                        },
                                        "description": {
                                                "urls": []
                                        }
                                },
                                "protected": false,
                                "followers_count": 2566921,
                                "friends_count": 379,
                                "listed_count": 15065,
                                "created_at": "Sat Jan 31 03:19:43 +0000 2009",
                                "favourites_count": 1281,
                                "utc_offset": -32400,
                                "time_zone": "Alaska",
                                "geo_enabled": false,
                                "verified": true,
                                "statuses_count": 6328,
                                "lang": "en",
                                "contributors_enabled": false,
                                "is_translator": false,
                                "is_translation_enabled": false,
                                "profile_background_color": "0B090B",
                                "profile_background_image_url": "http://pbs.twimg.com/profile_background_images/8479565/twitter_jpl_bkg.009.jpg",
                                "profile_background_image_url_https": "https://pbs.twimg.com/profile_background_images/8479565/twitter_jpl_bkg.009.jpg",
                                "profile_background_tile": false,
                                "profile_image_url": "http://pbs.twimg.com/profile_images/2305452633/lg0hov3l8g4msxbdwv48_normal.jpeg",
                                "profile_image_url_https": "https://pbs.twimg.com/profile_images/2305452633/lg0hov3l8g4msxbdwv48_normal.jpeg",
                                "profile_banner_url": "https://pbs.twimg.com/profile_banners/19802879/1398298134",
                                "profile_link_color": "0D1787",
                                "profile_sidebar_border_color": "100F0E",
                                "profile_sidebar_fill_color": "74A6CD",
                                "profile_text_color": "0C0C0D",
                                "profile_use_background_image": true,
                                "has_extended_profile": false,
                                "default_profile": false,
                                "default_profile_image": false,
                                "following": null,
                                "follow_request_sent": null,
                                "notifications": null,
                                "translator_type": "none"
                        },
                        "geo": null,
                        "coordinates": null,
                        "place": null,
                        "contributors": null,
                        "is_quote_status": false,
                        "retweet_count": 325,
                        "favorite_count": 1280,
                        "favorited": false,
                        "retweeted": false,
                        "possibly_sensitive": false,
                        "lang": "en"
                }
        ],
        "search_metadata": {
                "completed_in": 0.057,
                "max_id": 0,
                "max_id_str": "0",
                "next_results": "?max_id=967574182522482687&q=nasa&include_entities=1&result_type=popular",
                "query": "nasa",
                "count": 3,
                "since_id": 0,
                "since_id_str": "0"
        }
}
1 Solution

Damien_Dallimor
Ultra Champion

Search answers , lots of answers already about plugging in a custom response handler for the Rest Modular Input that will perform preprocessing actions such as splitting out events.

Add a custom handler to rest_ta/bin/responsehandlers.py

class TwitterStatusHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for status in output["statuses"]:
                print_xml_stream(json.dumps(status))   
        else:
            print_xml_stream(raw_response_output)

Apply custom handler to your REST input

alt text

View solution in original post

Damien_Dallimor
Ultra Champion

Search answers , lots of answers already about plugging in a custom response handler for the Rest Modular Input that will perform preprocessing actions such as splitting out events.

Add a custom handler to rest_ta/bin/responsehandlers.py

class TwitterStatusHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if response_type == "json":        
            output = json.loads(raw_response_output)

            for status in output["statuses"]:
                print_xml_stream(json.dumps(status))   
        else:
            print_xml_stream(raw_response_output)

Apply custom handler to your REST input

alt text

celianouguier
Explorer

Thank you so much Damien, It works like a charm !

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...