APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues

[dfbugs]

Mantis version info:

MantisBT Version: 2.25.6
Schema Version: 211
PHP Version: 7.4.33
Database Driver: mysqli
Database Version, Description: 10.4.31, 10.4.31-MariaDB

The server is hosted on a cloud server running AlmaLinux 8 as the OS


We've recently been running into a problem with getting the below error message whenever a user is trying to view individual issues, or trying to report issues:

APPLICATION ERROR #401

Database query failed. Error received from database was #1: Can't create/write to file '/tmp/MYuBr9e3' (Errcode: 28 "No space left on device") for the query: SELECT id, name, description FROM mantis_tag_table WHERE id NOT IN (SELECT tag_id FROM mantis_bug_tag_table WHERE bug_id = ?) ORDER BY name ASC .

Please use the "Back" button in your web browser to return to the previous page. There you can correct whatever problems were identified in this error or select another action. You can also click an option from the menu bar to go directly to a new section.

From doing some searching I figured that the issue is caused by the database running out of room. So I increased the amount of storage through the cloud server host by 20GB. This did resolve the issue for a while, but after around 2 weeks it returned again, and I don't believe it's possible that all 20GB has been used up in that time period.

Some quirks that I have observed:
1. For some reason, the anonymous user can view individual issues with no problems without getting the error, but registered users (including admins) will receive the error instead
2. For both times this came up, the first time that I restarted the server temporarily resolves the issue, and issues can be viewed/reported without getting the error message. But within a few hours the error message comes back and there are no changes for any restarts afterwards.


I'm looking for help on any experience with this error, information on where the "/tmp/" directory is located (in case the added storage wasn't allocated to the proper location), or whether there is something in the /tmp/ directory that is continuously writing files without proper clean up

Thank you!

[atrol]

Don't you have shell / command line access to this server, so that you can check what is stored under /tmp?

[dfbugs]

Yes I do have shell access to the server, this is the result of looking in the /tmp/ directory from root

[root@localhost ~]# ls /tmp/
snap-private-tmp
systemd-private-745a6af2a63c4b6189085a07edf42019-chronyd.service-T2faDQ
systemd-private-745a6af2a63c4b6189085a07edf42019-httpd.service-35Rat0
systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3
vmware-root_935-3980298462
vmware-root_937-4013854423
vmware-root_948-2688554130
vmware-root_969-4281777807
vmware-root_975-4281646728
vmware-root_980-2957518026

However I'm not sure if this is the correct /tmp/ to be looking at

[atrol]

What's the output of the following commands?

df -h /tmp
du -a /tmp

[dfbugs]

[root@localhost ~]# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/almalinux-root 58G 58G 20K 100% /

[root@localhost ~]# du -a /tmp
0 /tmp/vmware-root_975-4281646728
0 /tmp/vmware-root_937-4013854423
0 /tmp/vmware-root_948-2688554130
0 /tmp/vmware-root_969-4281777807
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_58sd9bdk9v1286vbfsm9pfrpo1
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_8l0vrkjp2shd0f6uua8mh8upmd
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_vhf2shu84pn9k31q0da3vnk04h
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_sehn53785h1aub11leun0224j2
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_p0mmji9eob4j22onk01tm16ks5
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_h3ckm8mui99ioc086iiq8eevc3
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_bl1inkr3cbhjugbvvcsb1ndvk0
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_d9ihr6fv8drs0hjbrfn41le7gg
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_sgj2pm2sq5p7gpon9bif0a2i1r
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_9t0irl5ffuclk2icnb8ld0h1u6
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_kmefp4lc8ulnurt1j4ukmoje18
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_saqackg5n3bq1l12cjde934l0u
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_9oj3c5kf9t1f8fjd4br3ga7g9k
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_2o23vrhuopgpph5d8e6k4csd5n
60 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp
60 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3
0 /tmp/vmware-root_935-3980298462
0 /tmp/snap-private-tmp
0 /tmp/.X11-unix
0 /tmp/.ICE-unix
0 /tmp/.XIM-unix
0 /tmp/.font-unix
0 /tmp/.Test-unix
0 /tmp/vmware-root_980-2957518026
0 /tmp/.SharedMutex_Acronis Scheduler2 LogFlagsSharedMutex
0 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-chronyd.service-T2faDQ/tmp
0 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-chronyd.service-T2faDQ
0 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-httpd.service-35Rat0/tmp
0 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-httpd.service-35Rat0
64 /tmp

[atrol]

Next command
du -sk /* 2>/dev/null
Check for the biggest number in the first column,e.g. might be where the second column is /usr
After that, e.g. if it should be /usr
cd /usr
du -sk * 2>/dev/null

[dfbugs]

Code: Select all

[root@localhost ~]# du -sk /* 2>/dev/null
0       /bin
373736  /boot
4       /dev
27108   /etc
4       /fetch_query_2d67702d92277d4b8a3f2e89d38d964c.post
1621036 /home
0       /lib
0       /lib64
0       /media
0       /mnt
383784  /opt
0       /proc
127616  /root
98008   /run
0       /sbin
0       /snap
0       /srv
0       /sys
64      /tmp
3601220 /usr
54989389        /var

Code: Select all

[root@localhost usr]# du -sk * 2>/dev/null
239920  bin
0       games
30444   include
2279416 lib
361104  lib64
134596  libexec
36      local
70964   sbin
399752  share
84988   src
0       tmp

[atrol]

The biggest number was not /usr it was /var in your case, so
cd /var
du -sk * 2>/dev/null

[dfbugs]

Ah you're right, this is the result for /var instead

Code: Select all

[root@localhost usr]# cd /var
[root@localhost var]# du -sk * 2>/dev/null
0       adm
87240   cache
0       crash
0       db
0       empty
0       ftp
0       games
0       gopher
0       kerberos
2106705 lib
0       local
0       lock
52795024        log
0       mail
0       nis
0       opt
0       preserve
0       run
0       snap
412     spool
0       tmp
0       www
0       yp

[atrol]

cd /var/log
du
ls -l

[dfbugs]

Code: Select all

[root@localhost var]# cd /var/log
[root@localhost log]# du
0       ./private
28      ./sssd
60      ./tuned
32844   ./audit
0       ./chrony
0       ./php-fpm
52408036        ./httpd
4       ./mail
19688   ./Acronis/APL
19688   ./Acronis
1192    ./letsencrypt
52795024        .

Code: Select all

[root@localhost log]# ls -l
total 333184
drwxr-xr-x  3 root   root          17 Oct 16 10:11 Acronis
drwx------. 2 root   root          99 Jan  5 01:37 audit
-rw-------  1 root   root       79262 Feb  5 19:42 boot.log
-rw-------  1 root   root       68325 Nov  3 03:14 boot.log-20231103
-rw-------  1 root   root       44665 Nov  7 03:41 boot.log-20231107
-rw-------  1 root   root       29712 Nov  8 03:42 boot.log-20231108
-rw-------  1 root   root       14856 Nov  9 03:07 boot.log-20231109
-rw-------  1 root   root       15525 Nov 10 03:23 boot.log-20231110
-rw-------  1 root   root       96794 Jan 17 03:47 boot.log-20240117
-rw-rw----  1 root   utmp    77086720 Feb  6 20:14 btmp
-rw-rw----  1 root   utmp   146380416 Jan  1 03:24 btmp-20240101
drwxr-x---. 2 chrony chrony         6 Oct  8  2022 chrony
-rw-r--r--  1 root   root     1006842 Feb  5 19:42 cloud-init.log
-rw-r-----  1 root   adm        58033 Feb  5 19:42 cloud-init-output.log
-rw-------  1 root   root       20480 Feb  6 20:15 cron
-rw-------  1 root   root       48765 Dec 31 03:51 cron-20231231
-rw-------  1 root   root       43587 Jan 17 03:47 cron-20240117
-rw-------  1 root   root       27758 Jan 21 03:22 cron-20240121
-rw-------  1 root   root       45208 Jan 28 13:45 cron-20240128
-rw-r--r--  1 root   root      126976 Feb  6 17:46 dnf.librepo.log
-rw-r--r--. 1 root   root     1048507 Dec 28 12:13 dnf.librepo.log.1
-rw-r--r--  1 root   root      229376 Feb  6 17:46 dnf.log
-rw-r--r--  1 root   root     1048556 Dec 28 15:47 dnf.log.1
-rw-r--r--. 1 root   root     1048533 Nov 21 11:43 dnf.log.2
-rw-r--r--. 1 root   root      172032 Feb  6 17:46 dnf.rpm.log
-rw-r--r--  1 root   root           0 Feb  6 17:46 hawkey.log
-rw-r--r--  1 root   root        2760 Dec 17 02:59 hawkey.log-20231217
-rw-r--r--  1 root   root        2700 Dec 24 03:27 hawkey.log-20231224
-rw-r--r--  1 root   root        2580 Dec 31 01:36 hawkey.log-20231231
-rw-r--r--  1 root   root        3780 Jan  9 17:49 hawkey.log-20240117
drwx------  2 root   root        4096 Jan 28 13:45 httpd
-rw-r--r--  1 root   root         659 Feb 28  2023 kdump.log
-rw-rw-r--. 1 root   utmp      292584 Feb  6 19:38 lastlog
drwx------  2 root   root        8192 Jan 29 06:26 letsencrypt
drwxr-xr-x  2 root   root          32 Nov  9 15:26 mail
-rw-------  1 root   root           0 Jan 28 13:45 maillog
-rw-------  1 root   root           0 Dec 24 03:47 maillog-20231231
-rw-------  1 root   root           0 Dec 31 03:51 maillog-20240117
-rw-------  1 root   root           0 Jan 17 03:47 maillog-20240121
-rw-------  1 root   root           0 Jan 21 03:22 maillog-20240128
-rw-------  1 root   root     1351680 Feb  6 20:15 messages
-rw-------  1 root   root     1776268 Dec 31 03:51 messages-20231231
-rw-------  1 root   root     2500882 Jan 17 03:47 messages-20240117
-rw-------  1 root   root      974882 Jan 21 03:19 messages-20240121
-rw-------  1 root   root     1560104 Jan 28 13:45 messages-20240128
drwxrwx---  2 apache root           6 Apr  3  2023 php-fpm
-rw-------  1 root   root         290 Nov  2 18:41 php-fpm.log
drwx------. 2 root   root           6 May 24  2021 private
-rw-------  1 root   root     9687040 Feb  6 20:14 secure
-rw-------  1 root   root    33876931 Dec 31 03:50 secure-20231231
-rw-------  1 root   root    18409195 Jan 17 03:46 secure-20240117
-rw-------  1 root   root    17540550 Jan 21 03:22 secure-20240121
-rw-------  1 root   root    24120605 Jan 28 13:45 secure-20240128
-rw-------  1 root   root           0 Jan 28 13:45 spooler
-rw-------  1 root   root           0 Dec 24 03:47 spooler-20231231
-rw-------  1 root   root           0 Dec 31 03:51 spooler-20240117
-rw-------  1 root   root           0 Jan 17 03:47 spooler-20240121
-rw-------  1 root   root           0 Jan 21 03:22 spooler-20240128
drwxr-x---. 2 sssd   sssd         167 Nov  5 03:50 sssd
-rw-r--r--  1 root   root      173709 Nov 29 01:56 trueimage-setup.log
drwxr-xr-x. 2 root   root          23 Apr  1  2023 tuned
-rw-r--r--  1 root   root         241 Feb  5 19:42 vmware-network.1.log
-rw-r--r--  1 root   root         187 Feb  5 19:37 vmware-network.2.log
-rw-r--r--  1 root   root         241 Feb  5 19:36 vmware-network.3.log
-rw-r--r--  1 root   root         189 Jan 30 15:19 vmware-network.5.log
-rw-r--r--  1 root   root         189 Jan 30 05:48 vmware-network.7.log
-rw-r--r--  1 root   root         189 Jan 16 16:09 vmware-network.9.log
-rw-r--r--  1 root   root         187 Feb  5 19:42 vmware-network.log
-rw-------  1 root   root       40519 Feb  5 19:42 vmware-vgauthsvc.log.0
-rw-------  1 root   root      116785 Feb  6 12:56 vmware-vmsvc-root.log
-rw-------  1 root   root       15846 Feb  5 19:42 vmware-vmtoolsd-root.log
-rw-rw-r--. 1 root   utmp       81792 Feb  6 19:38 wtmp

[atrol]

So the largest folder is obviously /var/log/httpd where typically the web server logs are stored.
cd /var/log/httpd
ls -lt

[dfbugs]

Code: Select all

[root@localhost log]# cd /var/log/httpd
[root@localhost httpd]# ls -lt
total 52408032
-rw-r--r-- 1 root root  1541894144 Feb  6 21:29 default_error_log
-rw-r--r-- 1 root root      409600 Feb  6 21:29 default_log
-rw-r--r-- 1 root root        7745 Feb  5 19:42 error_log
-rw-r--r-- 1 root root       12335 Jan 28 13:45 error_log-20240128
-rw-r--r-- 1 root root 10396575607 Jan 28 13:44 default_error_log-20240128
-rw-r--r-- 1 root root     1301500 Jan 28 13:44 default_log-20240128
-rw-r--r-- 1 root root        7398 Jan 21 03:22 error_log-20240121
-rw-r--r-- 1 root root 17795010685 Jan 21 03:21 default_error_log-20240121
-rw-r--r-- 1 root root      636570 Jan 21 03:21 default_log-20240121
-rw-r--r-- 1 root root      738979 Jan 17 03:47 default_log-20240117
-rw-r--r-- 1 root root 14471850453 Jan 17 03:47 default_error_log-20240117
-rw-r--r-- 1 root root        8460 Jan 17 03:47 error_log-20240117
-rw-r--r-- 1 root root      826605 Dec 31 03:51 default_log-20231231
-rw-r--r-- 1 root root  9456440949 Dec 31 03:51 default_error_log-20231231
-rw-r--r-- 1 root root         814 Dec 31 03:51 error_log-20231231
-rw-r--r-- 1 root root           0 Nov  9 23:07 ssl_request_log
-rw-r--r-- 1 root root           0 Nov  9 23:07 ssl_access_log
-rw-r--r-- 1 root root           0 Nov  9 23:07 ssl_error_log
-rw-r--r-- 1 root root           0 Mar  1  2023 access_log

[atrol]

To free up some disk space run
rm default_error_log-2*

To find out what's going on run
tail default_error_log

Before pasting the result to the forum, ensure that there is no private information in it.

[dfbugs]

Removing some of the error logs did help make the insufficient space error message go away for now!

here's an example of the output from tail default_error_log, it keeps going on and concerningly the time of the messages are all dated for today, which means that these error messages are still ongoing.

Code: Select all

[Tue Feb 06 22:00:13.606494 2024] [proxy_fcgi:error] [pid xxxxxx:tid xxxxxxxxxxx
 xxxx] [client xx.xx.xx.xxx:xxxxx] AH01071: Got error 'PHP message: PHP Warning:
 Use of undefined constant LOG_EMAIL_REIPIENT - assumed 'LOG_EMAIL_REIPIENT' (th
 is will throw an Error in a future version of PHP) in /home/default/html/config/
 config_inc.php on line 29PHP message: PHP Warning:  A non-numeric value encounte
 red in /home/default/html/config/config_inc.php on line 29PHP message: 2024-02-0
 6 14:00 PST DB database_api.php:166 db_connect() array (\n  0 => 'SET NAMES UTF8
 ',\n  1 => '0.0001',\n)\nPHP message: 2024-02-06 14:00 PST DB plugin_api.php:100
 5 plugin_register_installed() array (\n  0 => 'SELECT basename, priority, protec
 ted \n\t\tFROM mantis_plugin_table \n\t\tWHERE enabled=\\'1\\'\n\t\tORDER BY pri
 ority DESC',\n  1 => '0.0003',\n)\nPHP message: 2024-02-06 14:00 PST DB config_a
 pi.php:744 config_cache_all() array (\n  0 => 'SELECT config_id, user_id, projec
 t_id, type,  value, access_reqd FROM mantis_config_table',\n  1 => '0.0004',\n)\
 nPHP message: 2024-02-06 14:00 PST DB user_api.php:704 user_get_id_by_name() arr
 ay (\n  0 => 'SELECT * FROM mantis_user_table WHERE username=\\'anonymous\\'',\n
 1 => '0.0003',\n)\nPHP message: 2024-02-06 14:00 PST DB user_pref_api.php:334                                                                              user_pref_cache_array_rows() array (\n  0 => 'SELECT * FROM mantis_user_pref_tab
 le WHERE user_id IN (7711) AND project_id = 0',\n  1 => '0.0012',\n)\nPHP messag
 e: 2024-02-06 14:00 PST DB user_api.php:130 user_cache_array_rows() array (\n  0
 => 'SELECT * FROM mantis_user_table WHERE id IN (14478)',\n  1 => '0.0002',\n)\
 n