Page 1 of 2
APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 05 Feb 2024, 20:35
by dfbugs
Mantis version info:
MantisBT Version: 2.25.6
Schema Version: 211
PHP Version: 7.4.33
Database Driver: mysqli
Database Version, Description: 10.4.31, 10.4.31-MariaDB
The server is hosted on a cloud server running AlmaLinux 8 as the OS
We've recently been running into a problem with getting the below error message whenever a user is trying to view individual issues, or trying to report issues:
APPLICATION ERROR #401
Database query failed. Error received from database was #1: Can't create/write to file '/tmp/MYuBr9e3' (Errcode: 28 "No space left on device") for the query: SELECT id, name, description FROM mantis_tag_table WHERE id NOT IN (SELECT tag_id FROM mantis_bug_tag_table WHERE bug_id = ?) ORDER BY name ASC .
Please use the "Back" button in your web browser to return to the previous page. There you can correct whatever problems were identified in this error or select another action. You can also click an option from the menu bar to go directly to a new section.
From doing some searching I figured that the issue is caused by the database running out of room. So I increased the amount of storage through the cloud server host by 20GB. This did resolve the issue for a while, but after around 2 weeks it returned again, and I don't believe it's possible that all 20GB has been used up in that time period.
Some quirks that I have observed:
1. For some reason, the anonymous user can view individual issues with no problems without getting the error, but registered users (including admins) will receive the error instead
2. For both times this came up, the first time that I restarted the server temporarily resolves the issue, and issues can be viewed/reported without getting the error message. But within a few hours the error message comes back and there are no changes for any restarts afterwards.
I'm looking for help on any experience with this error, information on where the "/tmp/" directory is located (in case the added storage wasn't allocated to the proper location), or whether there is something in the /tmp/ directory that is continuously writing files without proper clean up
Thank you!
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 08:08
by atrol
Don't you have shell / command line access to this server, so that you can check what is stored under /tmp?
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 16:09
by dfbugs
Yes I do have shell access to the server, this is the result of looking in the /tmp/ directory from root
[root@localhost ~]# ls /tmp/
snap-private-tmp
systemd-private-745a6af2a63c4b6189085a07edf42019-chronyd.service-T2faDQ
systemd-private-745a6af2a63c4b6189085a07edf42019-httpd.service-35Rat0
systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3
vmware-root_935-3980298462
vmware-root_937-4013854423
vmware-root_948-2688554130
vmware-root_969-4281777807
vmware-root_975-4281646728
vmware-root_980-2957518026
However I'm not sure if this is the correct /tmp/ to be looking at
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 18:24
by atrol
What's the output of the following commands?
df -h /tmp
du -a /tmp
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 18:55
by dfbugs
[root@localhost ~]# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/almalinux-root 58G 58G 20K 100% /
[root@localhost ~]# du -a /tmp
0 /tmp/vmware-root_975-4281646728
0 /tmp/vmware-root_937-4013854423
0 /tmp/vmware-root_948-2688554130
0 /tmp/vmware-root_969-4281777807
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_58sd9bdk9v1286vbfsm9pfrpo1
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_8l0vrkjp2shd0f6uua8mh8upmd
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_vhf2shu84pn9k31q0da3vnk04h
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_sehn53785h1aub11leun0224j2
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_p0mmji9eob4j22onk01tm16ks5
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_h3ckm8mui99ioc086iiq8eevc3
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_bl1inkr3cbhjugbvvcsb1ndvk0
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_d9ihr6fv8drs0hjbrfn41le7gg
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_sgj2pm2sq5p7gpon9bif0a2i1r
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_9t0irl5ffuclk2icnb8ld0h1u6
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_kmefp4lc8ulnurt1j4ukmoje18
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_saqackg5n3bq1l12cjde934l0u
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_9oj3c5kf9t1f8fjd4br3ga7g9k
4 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp/sess_2o23vrhuopgpph5d8e6k4csd5n
60 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3/tmp
60 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-php-fpm.service-WASoe3
0 /tmp/vmware-root_935-3980298462
0 /tmp/snap-private-tmp
0 /tmp/.X11-unix
0 /tmp/.ICE-unix
0 /tmp/.XIM-unix
0 /tmp/.font-unix
0 /tmp/.Test-unix
0 /tmp/vmware-root_980-2957518026
0 /tmp/.SharedMutex_Acronis Scheduler2 LogFlagsSharedMutex
0 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-chronyd.service-T2faDQ/tmp
0 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-chronyd.service-T2faDQ
0 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-httpd.service-35Rat0/tmp
0 /tmp/systemd-private-745a6af2a63c4b6189085a07edf42019-httpd.service-35Rat0
64 /tmp
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 19:18
by atrol
Next command
du -sk /* 2>/dev/null
Check for the biggest number in the first column,e.g. might be where the second column is /usr
After that, e.g. if it should be /usr
cd /usr
du -sk * 2>/dev/null
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 19:42
by dfbugs
Code: Select all
[root@localhost ~]# du -sk /* 2>/dev/null
0 /bin
373736 /boot
4 /dev
27108 /etc
4 /fetch_query_2d67702d92277d4b8a3f2e89d38d964c.post
1621036 /home
0 /lib
0 /lib64
0 /media
0 /mnt
383784 /opt
0 /proc
127616 /root
98008 /run
0 /sbin
0 /snap
0 /srv
0 /sys
64 /tmp
3601220 /usr
54989389 /var
Code: Select all
[root@localhost usr]# du -sk * 2>/dev/null
239920 bin
0 games
30444 include
2279416 lib
361104 lib64
134596 libexec
36 local
70964 sbin
399752 share
84988 src
0 tmp
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 19:51
by atrol
The biggest number was not /usr it was /var in your case, so
cd /var
du -sk * 2>/dev/null
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 19:58
by dfbugs
Ah you're right, this is the result for /var instead
Code: Select all
[root@localhost usr]# cd /var
[root@localhost var]# du -sk * 2>/dev/null
0 adm
87240 cache
0 crash
0 db
0 empty
0 ftp
0 games
0 gopher
0 kerberos
2106705 lib
0 local
0 lock
52795024 log
0 mail
0 nis
0 opt
0 preserve
0 run
0 snap
412 spool
0 tmp
0 www
0 yp
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 20:08
by atrol
cd /var/log
du
ls -l
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 20:16
by dfbugs
Code: Select all
[root@localhost var]# cd /var/log
[root@localhost log]# du
0 ./private
28 ./sssd
60 ./tuned
32844 ./audit
0 ./chrony
0 ./php-fpm
52408036 ./httpd
4 ./mail
19688 ./Acronis/APL
19688 ./Acronis
1192 ./letsencrypt
52795024 .
Code: Select all
[root@localhost log]# ls -l
total 333184
drwxr-xr-x 3 root root 17 Oct 16 10:11 Acronis
drwx------. 2 root root 99 Jan 5 01:37 audit
-rw------- 1 root root 79262 Feb 5 19:42 boot.log
-rw------- 1 root root 68325 Nov 3 03:14 boot.log-20231103
-rw------- 1 root root 44665 Nov 7 03:41 boot.log-20231107
-rw------- 1 root root 29712 Nov 8 03:42 boot.log-20231108
-rw------- 1 root root 14856 Nov 9 03:07 boot.log-20231109
-rw------- 1 root root 15525 Nov 10 03:23 boot.log-20231110
-rw------- 1 root root 96794 Jan 17 03:47 boot.log-20240117
-rw-rw---- 1 root utmp 77086720 Feb 6 20:14 btmp
-rw-rw---- 1 root utmp 146380416 Jan 1 03:24 btmp-20240101
drwxr-x---. 2 chrony chrony 6 Oct 8 2022 chrony
-rw-r--r-- 1 root root 1006842 Feb 5 19:42 cloud-init.log
-rw-r----- 1 root adm 58033 Feb 5 19:42 cloud-init-output.log
-rw------- 1 root root 20480 Feb 6 20:15 cron
-rw------- 1 root root 48765 Dec 31 03:51 cron-20231231
-rw------- 1 root root 43587 Jan 17 03:47 cron-20240117
-rw------- 1 root root 27758 Jan 21 03:22 cron-20240121
-rw------- 1 root root 45208 Jan 28 13:45 cron-20240128
-rw-r--r-- 1 root root 126976 Feb 6 17:46 dnf.librepo.log
-rw-r--r--. 1 root root 1048507 Dec 28 12:13 dnf.librepo.log.1
-rw-r--r-- 1 root root 229376 Feb 6 17:46 dnf.log
-rw-r--r-- 1 root root 1048556 Dec 28 15:47 dnf.log.1
-rw-r--r--. 1 root root 1048533 Nov 21 11:43 dnf.log.2
-rw-r--r--. 1 root root 172032 Feb 6 17:46 dnf.rpm.log
-rw-r--r-- 1 root root 0 Feb 6 17:46 hawkey.log
-rw-r--r-- 1 root root 2760 Dec 17 02:59 hawkey.log-20231217
-rw-r--r-- 1 root root 2700 Dec 24 03:27 hawkey.log-20231224
-rw-r--r-- 1 root root 2580 Dec 31 01:36 hawkey.log-20231231
-rw-r--r-- 1 root root 3780 Jan 9 17:49 hawkey.log-20240117
drwx------ 2 root root 4096 Jan 28 13:45 httpd
-rw-r--r-- 1 root root 659 Feb 28 2023 kdump.log
-rw-rw-r--. 1 root utmp 292584 Feb 6 19:38 lastlog
drwx------ 2 root root 8192 Jan 29 06:26 letsencrypt
drwxr-xr-x 2 root root 32 Nov 9 15:26 mail
-rw------- 1 root root 0 Jan 28 13:45 maillog
-rw------- 1 root root 0 Dec 24 03:47 maillog-20231231
-rw------- 1 root root 0 Dec 31 03:51 maillog-20240117
-rw------- 1 root root 0 Jan 17 03:47 maillog-20240121
-rw------- 1 root root 0 Jan 21 03:22 maillog-20240128
-rw------- 1 root root 1351680 Feb 6 20:15 messages
-rw------- 1 root root 1776268 Dec 31 03:51 messages-20231231
-rw------- 1 root root 2500882 Jan 17 03:47 messages-20240117
-rw------- 1 root root 974882 Jan 21 03:19 messages-20240121
-rw------- 1 root root 1560104 Jan 28 13:45 messages-20240128
drwxrwx--- 2 apache root 6 Apr 3 2023 php-fpm
-rw------- 1 root root 290 Nov 2 18:41 php-fpm.log
drwx------. 2 root root 6 May 24 2021 private
-rw------- 1 root root 9687040 Feb 6 20:14 secure
-rw------- 1 root root 33876931 Dec 31 03:50 secure-20231231
-rw------- 1 root root 18409195 Jan 17 03:46 secure-20240117
-rw------- 1 root root 17540550 Jan 21 03:22 secure-20240121
-rw------- 1 root root 24120605 Jan 28 13:45 secure-20240128
-rw------- 1 root root 0 Jan 28 13:45 spooler
-rw------- 1 root root 0 Dec 24 03:47 spooler-20231231
-rw------- 1 root root 0 Dec 31 03:51 spooler-20240117
-rw------- 1 root root 0 Jan 17 03:47 spooler-20240121
-rw------- 1 root root 0 Jan 21 03:22 spooler-20240128
drwxr-x---. 2 sssd sssd 167 Nov 5 03:50 sssd
-rw-r--r-- 1 root root 173709 Nov 29 01:56 trueimage-setup.log
drwxr-xr-x. 2 root root 23 Apr 1 2023 tuned
-rw-r--r-- 1 root root 241 Feb 5 19:42 vmware-network.1.log
-rw-r--r-- 1 root root 187 Feb 5 19:37 vmware-network.2.log
-rw-r--r-- 1 root root 241 Feb 5 19:36 vmware-network.3.log
-rw-r--r-- 1 root root 189 Jan 30 15:19 vmware-network.5.log
-rw-r--r-- 1 root root 189 Jan 30 05:48 vmware-network.7.log
-rw-r--r-- 1 root root 189 Jan 16 16:09 vmware-network.9.log
-rw-r--r-- 1 root root 187 Feb 5 19:42 vmware-network.log
-rw------- 1 root root 40519 Feb 5 19:42 vmware-vgauthsvc.log.0
-rw------- 1 root root 116785 Feb 6 12:56 vmware-vmsvc-root.log
-rw------- 1 root root 15846 Feb 5 19:42 vmware-vmtoolsd-root.log
-rw-rw-r--. 1 root utmp 81792 Feb 6 19:38 wtmp
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 20:42
by atrol
So the largest folder is obviously /var/log/httpd where typically the web server logs are stored.
cd /var/log/httpd
ls -lt
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 21:30
by dfbugs
Code: Select all
[root@localhost log]# cd /var/log/httpd
[root@localhost httpd]# ls -lt
total 52408032
-rw-r--r-- 1 root root 1541894144 Feb 6 21:29 default_error_log
-rw-r--r-- 1 root root 409600 Feb 6 21:29 default_log
-rw-r--r-- 1 root root 7745 Feb 5 19:42 error_log
-rw-r--r-- 1 root root 12335 Jan 28 13:45 error_log-20240128
-rw-r--r-- 1 root root 10396575607 Jan 28 13:44 default_error_log-20240128
-rw-r--r-- 1 root root 1301500 Jan 28 13:44 default_log-20240128
-rw-r--r-- 1 root root 7398 Jan 21 03:22 error_log-20240121
-rw-r--r-- 1 root root 17795010685 Jan 21 03:21 default_error_log-20240121
-rw-r--r-- 1 root root 636570 Jan 21 03:21 default_log-20240121
-rw-r--r-- 1 root root 738979 Jan 17 03:47 default_log-20240117
-rw-r--r-- 1 root root 14471850453 Jan 17 03:47 default_error_log-20240117
-rw-r--r-- 1 root root 8460 Jan 17 03:47 error_log-20240117
-rw-r--r-- 1 root root 826605 Dec 31 03:51 default_log-20231231
-rw-r--r-- 1 root root 9456440949 Dec 31 03:51 default_error_log-20231231
-rw-r--r-- 1 root root 814 Dec 31 03:51 error_log-20231231
-rw-r--r-- 1 root root 0 Nov 9 23:07 ssl_request_log
-rw-r--r-- 1 root root 0 Nov 9 23:07 ssl_access_log
-rw-r--r-- 1 root root 0 Nov 9 23:07 ssl_error_log
-rw-r--r-- 1 root root 0 Mar 1 2023 access_log
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 21:52
by atrol
To free up some disk space run
rm default_error_log-2*
To find out what's going on run
tail default_error_log
Before pasting the result to the forum, ensure that there is no private information in it.
Re: APPLICATION ERROR #401 (Errcode: 28 "No space left on device") when attempting to view or report issues
Posted: 06 Feb 2024, 22:22
by dfbugs
Removing some of the error logs did help make the insufficient space error message go away for now!
here's an example of the output from tail default_error_log, it keeps going on and concerningly the time of the messages are all dated for today, which means that these error messages are still ongoing.
Code: Select all
[Tue Feb 06 22:00:13.606494 2024] [proxy_fcgi:error] [pid xxxxxx:tid xxxxxxxxxxx
xxxx] [client xx.xx.xx.xxx:xxxxx] AH01071: Got error 'PHP message: PHP Warning:
Use of undefined constant LOG_EMAIL_REIPIENT - assumed 'LOG_EMAIL_REIPIENT' (th
is will throw an Error in a future version of PHP) in /home/default/html/config/
config_inc.php on line 29PHP message: PHP Warning: A non-numeric value encounte
red in /home/default/html/config/config_inc.php on line 29PHP message: 2024-02-0
6 14:00 PST DB database_api.php:166 db_connect() array (\n 0 => 'SET NAMES UTF8
',\n 1 => '0.0001',\n)\nPHP message: 2024-02-06 14:00 PST DB plugin_api.php:100
5 plugin_register_installed() array (\n 0 => 'SELECT basename, priority, protec
ted \n\t\tFROM mantis_plugin_table \n\t\tWHERE enabled=\\'1\\'\n\t\tORDER BY pri
ority DESC',\n 1 => '0.0003',\n)\nPHP message: 2024-02-06 14:00 PST DB config_a
pi.php:744 config_cache_all() array (\n 0 => 'SELECT config_id, user_id, projec
t_id, type, value, access_reqd FROM mantis_config_table',\n 1 => '0.0004',\n)\
nPHP message: 2024-02-06 14:00 PST DB user_api.php:704 user_get_id_by_name() arr
ay (\n 0 => 'SELECT * FROM mantis_user_table WHERE username=\\'anonymous\\'',\n
1 => '0.0003',\n)\nPHP message: 2024-02-06 14:00 PST DB user_pref_api.php:334 user_pref_cache_array_rows() array (\n 0 => 'SELECT * FROM mantis_user_pref_tab
le WHERE user_id IN (7711) AND project_id = 0',\n 1 => '0.0012',\n)\nPHP messag
e: 2024-02-06 14:00 PST DB user_api.php:130 user_cache_array_rows() array (\n 0
=> 'SELECT * FROM mantis_user_table WHERE id IN (14478)',\n 1 => '0.0002',\n)\
n