Why is pgrade Failing with OSError type 28?

[edwardrose]

Hello All,

I am testing the upgrade from ES 6.2.0 to 6.6.2.  When I do the upgrade it fails with OSError type 28 no space left of device.  But there is almost 30GB of disk space free.

2021-10-04 19:18:28,028 INFO    [615bb5deed7f2dc4595650] _cplogging:216 - [04/Oct/2021:19:18:28] HTTP
Request Headers:
  Remote-Addr: 127.0.0.1
  TE: chunked
  HOST: splunk-sh1.wv.mentorg.com:8000
  ACCEPT-ENCODING: gzip, br
  CACHE-CONTROL: max-age=0
  SEC-CH-UA: "Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93"
  SEC-CH-UA-MOBILE: ?0
  SEC-CH-UA-PLATFORM: "Windows"
  UPGRADE-INSECURE-REQUESTS: 1
  ORIGIN: null
  USER-AGENT: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36
  ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
  SEC-FETCH-SITE: same-origin
  SEC-FETCH-MODE: navigate
  SEC-FETCH-USER: ?1
  SEC-FETCH-DEST: document
  ACCEPT-LANGUAGE: en-US,en;q=0.9
  COOKIE: splunkweb_csrf_token_8000=[REDACTED]5649; session_id_8000=[REDACTED]5b74; token_key=[REDACTED]5649; experience_id=[REDACTED]b0c2; splunkd_8000=[REDACTED]tgchx
  REMOTE-USER: admin
  X-SPLUNKD: SKdIpkhtf8PlfUDwvOLunA== 11626949294704615649 ijbs1HY^4Ms541EE5sF6eqHg^iyD5t6QKZRByWhdMDXkj546^eB1lT6y59b9LewgHbLcz0Xa5SKotHijcl__zWhYqh8MZISrCqYVxuLkY7jijwyyXijSUQ9VAJRlcQA3o7tgchx 0
  Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryO0HdVIPxgJr5HUZN
  Content-Length: 675766277
2021-10-04 19:18:28,029 INFO    [615bb5deed7f2dc4595650] error:333 - POST /en-US/manager/appinstall/_upload 127.0.0.1 8065
2021-10-04 19:18:28,029 INFO    [615bb5deed7f2dc4595650] error:334 - 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request.
2021-10-04 19:18:28,029 ERROR   [615bb5deed7f2dc4595650] error:335 - Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond
    self._do_respond(path_info)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 680, in _do_respond
    self.body.process()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 982, in process
    super(RequestBody, self).process()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 559, in process
    proc(self)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 225, in process_multipart_form_data
    process_multipart(entity)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 217, in process_multipart
    part.process()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 557, in process
    self.default_proc()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 717, in default_proc
    self.file = self.read_into_file()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 732, in read_into_file
    self.read_lines_to_boundary(fp_out=fp_out)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 702, in read_lines_to_boundary
    fp_out.write(line)
OSError: [Errno 28] No space left on device

As you can see there should be plenty of room for a 670MB upload

splunk@splunk-sh1:~/var/log/splunk> df -kh /opt/splunk
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/system-splunk   74G   44G   27G  63% /opt
splunk@splunk-sh1:~/var/log/splunk>

Web.conf

splunk@splunk-sh1:~/var/log/splunk> more ~/etc/system/local/web.conf
[settings]
login_content = <h1> <CENTER>Splunk Dev Search Head</CENTER> </h1>
max_upload_size = 1024
enableSplunkWebSSL = 1
privKeyPath = /opt/splunk/etc/auth/splunkweb/com.key
caCertPath = /opt/splunk/etc/auth/splunkweb/expJun2022.crt
splunkdConnectionTimeout = 1400
tools.sessions.timeout = 180
sslVersions = ssl3,tls
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

splunk@splunk-sh1:~/var/log/splunk>

So I am confused why it would say that there is no space left of the device.

Thanks

ed

[edwardrose · Accepted solution]

I found this article

https://community.splunk.com/t5/Getting-Data-In/Tutorial-data-upload-error/m-p/214891

I reduce the setting as stated to 4800 and the upgrade proceeded just fine.

Weird.

[smithy001]

Had the same issue.

Cleared /tmp down and worked fine...

Can only assume splunk GUI loads to /tmp

[computermathguy]

We managed to resolve the the "type 28 / 500 internal server" Enterprise Security installation error by cleaning out /tmp.