Hey @padresman  Will try your example. Gotta be very careful that your expression fields match the capture group you use, as it will store it in "attributes."capture group value" by default.  Also, make sure to use golang regex on regex101. though your regex appears to be fine.  Also its wise to iterate and NOT remove the fields you make to see what they look like when they arrive at splunk. Can help make sure your value is what you think it is..... ... View more

All these questions can be answered in the cloud monitoring console and you should start there instead of trying to write your own bespoke spl. License > Storage overview is a great place to start. There is also DDAS and DDAA searches there. specifically for archive data please review here as per docs: https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Admin/DataArchiver Steps to review the overall size and growth of your archived indexes You might want to review the size and growth of your archived indexes to better understand how much of your entitlement you are consuming. This can help you predict usage and expenses for your archived data. From Splunk Web, go to Settings > Indexes. From the Indexes page, click on a value in the Archive Retention column.     ... View more

No. I suggest follow up with your account team so they can see what values you are comparing and ensure they are accurate. (Or post what actual values you are looking at. All your storage forecasting should be is how much RAW data do you ingest a day, mulltipled by how many days you want to store in searchable + archive,  and does that fit into your DDAS and DDAA entitlement? Splunk Cloud service takes care of the rest.  No compression math at all vs on-prem days.   ... View more

I think u mean DDAA (dynamic data active archive) is archive storage (cold storage), and no we don't only store compressed raw data.  Whole bucket goes to archive storage then us copied back into object store upon restore. DDAS (dynamic data active searchable) is basically "smartstore".  Customers using ddas and ddaa only need to care about the raw size, not compression. Compression doesnt come into play with splunk cloud entitlements, so the old onprem math doesnt apply. now depending on what data hes actually looking at, he may or may not be seeing compressed buckets, but more often it comes down to understanding when buckets will actually roll due to timestamps in the bucket. overall in splunk cloud all you care about is raw data size as you are not sizing disk in cloud, u are sizing your subscription. https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Service/SplunkCloudservice see storage section ... View more

dev docs have been nicely updated over the last little while! shout to tedd! https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/secretstorage Theres API and SDK examples, and a nice post on how to control secret access, which has gotten better, and could still be better with more ppl pushing on it.  https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/secretstorage/secretstoragerbac your app just needs the proper role and capabilities to interact with the storage endpoint and can be scoped further from there.  ... View more

Hi! you'll want to poke thru the docs on splunk config files: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/AboutConfigurationFiles But tldr is I would use "btool" - https://docs.splunk.com/Documentation/Splunk/9.1.2/Troubleshooting/Usebtooltotroubleshootconfigurations and you'll want to go hunting for "inputs.conf" - this is where your spunk instances would be taking the data in, then comb thru props.conf - where the sourcetypes and event parsing/transformation/routing happens.. It is also common to have splunk co-located with a syslog listener who puts logs down that we pick up. So a quick `ss -tulpn` or `netstat -tulpn` will show what ports, if any, are open on your Heavy Forwarders.  so getting good with btool or reviewing Inputs and sourcetypes in your splunk ui will be key ... View more

Hi! Can you clarify what the actual source of these logs are? Are these logs in the pod's stdout/stderr logs from /var/log/pods, or are they in some other location? What is the path to them? Nix nodes or windows nodes? As mentioned already, OTel does fingerprint logs and filelog settings expect to be tailing live logs, so if this is a bit different use case we can add a custom receiver in the "extraFilelogs" section of the helm chart.  ... View more

Hey! Yeah OTel file exporter can be used.  https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/fileexporter You could also configure otel to filter the data you want and send to file and hec simultaneously! https://github.com/signalfx/splunk-otel-collector/tree/main/examples/otel-logs-routing Splunk otel distro has these components: https://github.com/signalfx/splunk-otel-collector/blob/main/docs/components.md ... View more

👋  ... View more

Last I checked it is still the case that all apps have access to the secret store and have to use controls to limit access.  Came up when working with 1Password on their app. Will poke around and see if any hardening or rework is in the cards this year.... Dev docs have seem to have been updated. https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/secretstorage/secretstoragerbac/#Configure-role-based-access-to-secret-storage If still think more is needed, hit the feedback link on this doc. dev team has heard this before. glad they have added the docs, but if still need more, good place to start.  ... View more

Maybe try and check that you define `$SPLUNK_HOME` environment variable correctly or just point it to the absolute path? Doest like your "bucket path" ... View more

This would heavily depend on what your events look like as you would simply extract fields that represent "remote" and "local" fields.  Got an example event? ... View more

So the issue you are having is the index it lands in correct? It is likely that SC4S hec client is sending a default index.  Hec clients settings in the payload override the settings you put on the token. Think of those as "if not set by the hec client".  https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/FormateventsforHTTPEventCollector#Event_metadata Would check sc4s docs on setting indexes: https://splunk.github.io/splunk-connect-for-syslog/main/configuration/#log-path-overrides-of-index-or-metadata     ... View more

"should I do in chunk"? - Yes, use the date ranges to reduce your date range and restore in multiple chunks.  No it will not "reindex it" - https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Admin/DataArchiver#Restore_archived_data_to_Splunk_Cloud_Platform You can use the "check size" button to make sure your span is under your entitlement. Remember Dynamic Data Active Archive (DDAA)  it is 10% of your Dynamic Data Active Searchable (DDAS), NOT your daily ingest entitlement. Check "cloud monitoring console> license usage > storage summary" Span too wide! too many buckets!: shorten the span, now i can restore!:   reduce your chunk size to under your limit, restore that data, search it, then in the table below you can clear it and restore you next chunk.  Data quality matters here, as if your timestamps are all over the place it can be suprizing how many buckets you have to restore to bring back any give date.  it will not take multiple days to restore this. if you just shrink your window you can do it in steps.  restore > search (tip use collect command to help move what you want to another index) > clear restore > repeat ... View more

Cloud monitoring console should provide a great start on analyzing your storage needs.  https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Admin/MonitoringLicenseUsage#Monitor_the_Storage_Summary_dashboard The key concept you must be familiar with, is the Splunk bucket lifecycle, as buckets are the smallest form of storage in Splunk, and impacts greatly how and when your buckets move from active searchable to active archive.  I wouldnt over complicate it with compression. While Splunk does compress data, your entitlements are on raw data ingested, so just closely analyze your daily ingest in your biggest indexes and poke around with the `dbinspect` command and the monitoring console to ensure your bucket health is good.  Data onboarding and data quality is key to ensure your timestamps dont pollute your buckets with timestamps way in the past or future, because a bucket can only migrate to archive when ALL EVENTS in the bucket meet the time/size criteria. https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Admin/MonitoringHealth#Health_indicator_information_and_additional_resources:~:text=Search%20Manual.-,Bucket%20size%20and%20range,-An%20index%20typically https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Admin/MonitoringIndexing#Verify_data_quality Also even going back and reading Splunk Enterprise docs on "smartstore" will help provide you with some good background, or work with your account team to go through it and ensure you have a good handle on it.  ... View more

1. Is there a way to directly install Custom Apps / Add-ons (that are originally built for Splunk Enterprise), in Splunk Cloud? We were thinking about compatibility issues, and if the apps would work the same way.  Yes, see "installing Private apps" on Splunk Cloud Platform. -https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Admin/PrivateApps 2.  Is there a way to gauge whether or not the quantity of data that we want to send from external sources, would require us to install a Heavy / Universal Forwarder? (We are trying to avoid additional costs by taking Splunk Cloud, so we were wondering if we could do without them) As long as your cloud deployment is sized correctly around how much ingest and search you plan to do, you can absolutley use cloud without the need for HFs or on-premesis infra. It's always an option when needed.  The main place to get familiar with is the Splunk Cloud Service Description. It lays out the service and any limits or recommends we have. For example,  https://docs.splunk.com/Documentation/SplunkCloud/9.1.2308/Service/SplunkCloudservice#Experience_designations       Victoria Classic Modular and scripted inputs Modular and scripted inputs can now run directly on the search tier without the additional overhead of a separate IDM instance. Review pull based service limits below: Up to 500GB/day for entitlement of less than 166 SVC or 1 TB Up to 1.5TB/day for more than 166 SVC or 1 TB Modular and scripted inputs must run on a separate IDM instance or customer-managed heavy forwarder. Victoria runs the inputs on the SH tier to allow self service. Classic runs the "HF"s for you as "IDM"s, but way less self service. So depends on what you value more.  The free cloud trial instances wont be what you want for actual testing etc. Have your Sales Engineer spin up a demo stack internally and you can play with them or do a full blown POC.  How much ingest do you plan to do? Check out the Splunk Cloud Migration Assessment app for help translating requirements.  https://splunkbase.splunk.com/app/4974 Hope that helps! Feel free to join others on splunk cloud in the splunk_cloud room on community slack too! splk.it/slack ... View more

The way to go is with the OpenTelemetry Helm Chart. Wrote a lil quickstart here https://github.com/matthewmodestino/otel-quickstart/blob/main/kubernetes/0-quickstart-home.md#kubernetes-otel-quickstart-home See docs and validated architecture for more! https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/OtelCollectorKubernetes https://docs.splunk.com/Documentation/SVA/current/Architectures/OTelKubernetes If you run into issues reach out to your SE, we have workshops or jump into the community slack channel splk.it/slack! holler at me in the kubernetes channel, or opentelemetry channels, (mattymo) ... View more

Yes, it would be specific to HEC clients that check for the endpoint's availability with tcp connections but not sending data.  This would not happen with HF because HTTP traffic would come in HF then be sent via S2S protocol to Indexers, which wouldnt do the checks like that.  sorry for the answer from far in the future 🙂  ... View more

Hello from the future! I ran into this exact thing working with OpenTelemetry's HEC exporter.  It turns out if the "raw" field contains a "\n" it will increment the linecount to 2.  This is likely because Splunk's default line breaker behaviour is to drop any new lines as part of the capture group it uses "([\r\n]+)" so the logic assumes new lines would be removed. so if it somehow sneaks into raw (in my case it skips linebreaking due to hec event endpoint)  even tho you dont see it in the raw even on screen it looks to trigger the linecount to be 2.      ... View more

Yeah, I linked that in my original answer. Let us know when you try and how it goes! ... View more

Not yet, sorry, will have to try next week.  ... View more

I think it will just use reload. will try and replicate in my stack when i get the chance and let you know. Docs say it should just be a reload automagically for props/tranforms changes.  ... View more

I wouldn't give up yet...we have a lot of behind the scenes logic that takes care of distributing your configs in cloud and reducing the need for restarts, and the _reload option is available in cloud...are you certain you need a refresh? Generally props, i dont believe should require them anymore... https://docs.splunk.com/Documentation/SplunkCloud/9.0.2303/Admin/RollingRestart#Reload_or_restart_behavior_of_common_.conf_files https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/Configurationfilechangesthatrequirerestart#Index-time_settings are you hitting some sort of issue or error? I can always test your app in my stack if you want... ... View more

Splunk can definitely what you are trying to do, maybe in too many ways :).  I think finding the one that works for you is the key and I think you are close in your options 1 & 2.  (I'd ditch #3.) option #1 this is probably the simplest way.  I'd start here.  I am not sure I grok why the source field is a problem with using the data, but it can definitely be replaced using props/transforms versus hardcoding in the inputs. This should allow Splunk to use it to grab your desired hostname with "host_regex" in inputs.conf, then we can overwrite it in a props/transforms as the data flows through the pipeline. I suggest simply using an ingest eval stanza like:   ## props.conf [my:sourcetype] TRANSFORMS = source_override ## transforms.conf [source_override] INGEST_EVAL = source:="my_simple_source"   Option #2  This may provide even more flexibility in your logic for setting the fields you want. I would also use  "ingest_eval" here as well versus the general props/transforms as it provides some powerful logic that regex alone may not be suitable for.  https://docs.splunk.com/Documentation/Splunk/9.0.4/Data/IngestEval I believe the issue you may be hitting with losing the hostname is that your current approach relies on the hostname being present in the raw data, which may not happen when linebroken.  Instead with ingest_eval we can write some logic that allows us to manipulate the metadata you get to achieve your goal.  Technically you would just be re-implementing the "host_regex" logic here tho, so might be overkill, but might be useful if needed for advanced uses and wouldn't be limited to only the source field.    ## props.conf [my:sourcetype] TRANSFORMS = extract_host_from_source,source_override ## transforms.conf [extract_host_from_source] SOURCE_KEY = MetaData:Source REGEX = <some_regex_that_extracts_your_host_value> FORMAT = host::$1 DEST_KEY = MetaData:Host # once you have the hostname from source, now overwrite it! [source_override] INGEST_EVAL = source:="my_simple_source"   tldr: ingest_eval is mad powerful and allows you to refer to fields or metadata that exists already and apply powerful eval logic that goes well beyond what I showed here. I've even done conditional field overrides based on fields extracted, etc. It's super cool and leads into ingest actions world eventually.  Note: I wrote this pseudocode style, haven't tested them against data. If you have some sample data happy to try and tune them if you run into issues.  References: https://docs.splunk.com/Documentation/Splunk/9.0.4/Data/Overridedefaulthostassignments https://docs.splunk.com/Documentation/Splunk/9.0.4/Data/IngestEval  https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/Transformsconf#:~:text=INGEST_EVAL%20%3D%20%3Ccomma%2Dseparated,Optional.%0A*%20Default%3A%20empty   ... View more