0035011: `tokenizer` php module is required, but not checked for and not documented as such

ID	Project	Category	View Status	Date Submitted	Last Update
0035011	mantisbt	installation	public	2024-11-20 16:01	2025-03-01 18:40

Reporter	rogueresearch	Assigned To	dregad
Priority	normal	Severity	minor	Reproducibility	have not tried
Status	closed	Resolution	fixed
Product Version	2.27.0
Target Version	2.27.1	Fixed in Version	2.27.1

Summary	0035011: `tokenizer` php module is required, but not checked for and not documented as such
Description	Today we got this error: `[2024-11-20 15:06:01.240638] [php:notice] [pid xxx:tid [client 10.x.x.x:xxx] Call to undefined function token_get_all()\n/usr/local/etc/apache24/mantis.example.com/core/classes/ConfigParser.class.php: 54: Tokenizer - -> - __construct()\n/usr/local/etc/apache24/mantis.example.com/adm_config_set.php: 81: ConfigParser - -> - __construct()\n, referer https://mantis.example.com/adm_config_page.php?user_id=0&project_id=95&config_option=bug_report_page_fields&action=clone` After debugging, we realized the issue was the PHP tokenizer module was required but not present. Installing it fixed the issue. But: The admin guide does mention this requirement Mantis' own self-check feature did not complain that the module was missing.
Tags	No tags attached.

dregad 2024-11-21 04:46 developer ~0069480	Thanks for the report, you're correct. Surprised that nobody's hit this since 2016 0020787 though... Anyway, will fix.

dregad 2024-11-21 07:10 developer ~0069481	Surprised that nobody's hit this since 2016 Looking at PHP documentation, this is most likely because the tokenizer extension is enabled by default, and built-in on Windows.

dregad 2024-11-21 12:53 developer ~0069483	PR https://github.com/mantisbt/mantisbt/pull/2049

rogueresearch 2024-11-21 13:28 reporter ~0069484	2016! Hmmm, it's suspicious that we ourselves did not run into it before now. We run mantis on FreeBSD. We recently updated PHP to 8.3. Perhaps the default PHP package build config used to enable `tokenizer` by default and no longer does. Anyway, your PR lgtm. Thanks!

MantisBT: master-2.27 c428605d 2024-11-21 07:22 dregad Details Diff	The tokenizer php extension is required Without it, adm_config_set.php throws a "Call to undefined function token_get_all()" error. - add the extension to composer.json - document the requirement in Admin Guide - verify its presence in admin checks Fixes 0035011	Affected Issues 0035011
	mod - admin/check/check_php_inc.php	Diff File
	mod - composer.json	Diff File
	mod - composer.lock	Diff File
	mod - docbook/Admin_Guide/en-US/Installation.xml	Diff File
MantisBT: master c428605d 2024-11-21 07:22 dregad Details Diff	The tokenizer php extension is required Without it, adm_config_set.php throws a "Call to undefined function token_get_all()" error. - add the extension to composer.json - document the requirement in Admin Guide - verify its presence in admin checks Fixes 0035011	Affected Issues 0035011
	mod - admin/check/check_php_inc.php	Diff File
	mod - composer.json	Diff File
	mod - composer.lock	Diff File
	mod - docbook/Admin_Guide/en-US/Installation.xml	Diff File