Deployment Architecture

Splunk is not indexing .gz files and returning gebrish results

aamer86
Path Finder

Hi,

I have some S3 access logs in S3 with .gz suffix which is not read by Splunk

I am using AWS Add-On to collect these logs using Incremental S3 option and I tried the general option

Two questions:

1- Can I using the incremental / generic S3 option blacklist any log file ending with .gz
2- Do I need to add anything within the AWS Add-On settings to allow Splunk to read the .GZ logs

I cant attached a photo of how logs appear in Splunk

dkr3500
Path Finder

I'm having this exact same issue.   I downloaded the file from S3 to my local computer, renamed it to have .gz extension, uploaded it back to S3, and Splunk was able to read it in clear text. 

But with that said, is there a way to let Splunk know that all my files in the bucket at zip files, but they don't have any file extensions at the end of the file?  This way, I know for a fact Splunk will read the files properly.

I'm using the Splunk Add-on for AWS to configure my inputs; Splunk Enterprise 7.2.4.

Thanks!

0 Karma

_joe
Communicator

Were you ever able to resolve this?

I am having a similar issue. Splunk is reading AWS s3 .gz files as "����" where as if I download the file to my Linux/HW I can use "gzip -d filename.gz" and it works just fine. As others suggested, it is a single file not a directory. 

0 Karma

ololdach
Builder

Hi,

please take a look at https://docs.splunk.com/Documentation/AddOns/released/AWS/S3 it says that splunk supports these file types for single files : ZIP, GZIP, TAR, or TAR.GZ and for multiple files/directories : ZIP, TAR, or TAR.GZ

The way I see it, '.gz' can only be used in single file inputs, not in multiple files. If you want to index multiple files, '.tgz' is supported.

hih
Oliver

woodcock
Esteemed Legend

That is probably it. Great point!

0 Karma

woodcock
Esteemed Legend

Out of the box, Splunk properly handles *.gz files so something must be very wonky with your configuration. You are going to have to give way more detail, especially your inputs.conf entry.

0 Karma

aamer86
Path Finder

[aws_s3://S3-AccessLogs]
aws_account = XXXXXXX
bucket_name = XXXXXXXX
character_set = auto
ct_blacklist = ^$
host_name = s3-eu-west-1.amazonaws.com
index = aws-cloudwatch
initial_scan_datetime = 2018-04-01T14:24:21Z
max_items = 100000
max_retries = 3
polling_interval = 1800
recursion_depth = -1
sourcetype = aws:s3:accesslogs
disabled = 1
crcSalt =

0 Karma

aamer86
Path Finder

�W EO� WFι��b61/�Z�Zt�ө����F������n8$�����1� ���R h8�����Wz���B 2Q�i4mf&;�N�9],��7���W &B�
��Tgp�/v��Äd�y����+ӁT��Ix��ˌ�ṱ�.��L�w��NH�$���<"���c����6�W�9����|֝ɺŅI��L!�:CI��dc ����������0�g�����L�w���9X�V�3$�����M6�Aó �Sp�?�K;~P����#m���C�ӓ
lp�I[��p�=��?("�Y*6���7?^��ts�����x�����C& � ΰNK6�"�R[n7�G�p`��I�R��p�a������q�\Ⱦ���A�6g4�tgY�$�W�v�c��K&{q��l@\�y�՗&&q���\t�6�{���d�-Dl����i��u@�A�g����ŕ���X;c
�G\'$��S�?t&q��l84 b�ww��Ђ��Ӂ��g��u�q�������z�x��
b���{2q��ld
g�8�]�\A�+��ciA�2���#��/�\��FA��Ǡ�>ϒ����Gl���X�i��b; ]��b�`�%)�B�g����FlI(�$������g/=8J����ң�O�F_��cm#�����ۏ>ܼ�E~�sM}ד�u�+d���X�

0 Karma

aamer86
Path Finder

this is how the logs are getting to Splunk

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...