View Issue Details

IDProjectCategoryView StatusLast Update
0032365mantisbtdb postgresqlpublic2023-04-28 14:06
Reportermpeterma Assigned Todregad  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionno change required 
PlatformServerOSNetBSDOS Version9.3
Product Version2.25.6 
Summary0032365: APPLICATION ERROR 401 when writing Custom Fields with more than 255 characters
Description

APPLICATION ERROR 401

Datenbankabfrage fehlgeschlagen. Die Reportmeldung lautet #-1: ERROR: value too long for type character varying(255) für die Abfrage: INSERT INTO mantis_bug_history_table
( user_id, bug_id, date_modified, field_name, old_value, new_value, type )
VALUES
( $1, $2, $3, $4, $5, $6, $7 ).
Bitte benutzen Sie die „Zurück“-Taste Ihres Browsers, um auf die vorhergehende Seite zurückzukehren. Dort können Sie den hier angezeigten Eintrag korrigieren oder eine andere Aktion ausführen. Über das Menü können Sie auch direkt zu einer anderen Aktion wechseln.

Steps To Reproduce

Steps:

  1. Define a custom field as text area
  2. Edit a bug ticket containing this field, add content with more than 255 characters
  3. Save the ticket

Result:

  • the error message appears
  • the content of the custom field is written
  • history records (old / new text) for the custom fields are missing
Additional Information

It looks like the character constraints in the mantis_bug_history_table (old_value / new_value columns) can't handle it when a custom field allows more than 255 characters. Restricting the custom field to a character length of 255 characters only helps me to a limited extent, because in this specific case I want to use the field to capture entries for a change log. Here 255 characters will usually not be enough.

Since the error message seems to come from the database, I'm not sure if this is possibly a particularly strict behavior of PostgreSQL (I'm using version 14) compared to other supported databases. As a hack, I could increase the varchar definitions for the above table fields in mantis_bug_history_table directly on the database. Before I do that, I would like to use this ticket to ask for an assessment if there might be a better way already now, or if it is not a Mantis bug at all.

TagsNo tags attached.

Relationships

related to 0025002 closedatrol Error when updating content in a custom field of type "Text Area" ("Textbereich"): History cannot be stored 
related to 0032385 closeddregad Incorrect use of mb_strimwidth() to truncate old/new values in history API 

Activities

dregad

dregad

2023-04-11 04:43

developer   ~0067623

Last edited: 2023-04-11 05:03

This looks like a duplicate of 0025002, which should be fixed since 2.19.0 by truncating the long string before recording it in history,

I don't have a working PostgreSQL setup available to test ATM to confirm, but at least on MySQL I'm not getting any error with the given repro steps. The long string is truncated to 252 chars and ... gets appended, insert into history table is successful.

EDIT: I have also confirmed that on MySQL I do get ERROR 1406 (22001): Data too long for column 'new_value' at row 1 if I (manually) try to insert data longer than 255 chars so the problem you're seeing is likely not PostgreSQL-specific.

I'm wondering if this could be due caused by the custom field data containing multibyte characters ?

dregad

dregad

2023-04-11 04:57

developer   ~0067624

Last edited: 2023-04-11 08:15

Actually looking at the patch MantisBT master 4609e811, problem might be caused by use of mb_strimwidth() function to truncate string, which I believe is not appropriate as width <> length.

Still @mpeterma it would help to get confirmation of the actual data you're trying to insert when the error occurs.

mpeterma

mpeterma

2023-04-11 13:09

reporter   ~0067634

Hello @dregad, thank you for the quick assessment and advice. You hit the nail on the head - the problem only occurs when multibyte characters appear in the string, such as the German umlauts üöäß. I have checked this again systematically in my installation with the following cases:

  1. string only ASCII characters 255 characters "012345678901234567890......". -> ok
  2. string only ASCII characters more than 255 characters "012345678901234567890......" -> ok, value is abbreviated in history entry as described with "..."
  3. string starting with multibyte "ä" + 255 more ASCII characters "012345678901234567890....". -> error occurs

Looking at the description of mb_strimwidth, it doesn't read wrong at first. I am thinking if it could also be because of how I created the database.... Normally I create my PostgreSQL databases with template1, which I assumed uses UTF-8 as encoding. However... I just let go of a "select character_set_name from information_schema.character_sets;" again on the affected DB and I get "SQL ASCII" as the result there. This looks strange to me. I'll have to check that again in more detail. If the DB is really created with ASCII encoding, then the 255 characters of the varchar correspond to only 255 real bytes. I will get back to you as soon as possible.

dregad

dregad

2023-04-11 18:21

developer   ~0067636

string starting with multibyte "ä" + 255 more ASCII characters "012345678901234567890....". -> error occurs

$s = 'ä' . str_repeat('a', 254);
echo strlen($s); // 256
echo mb_strlen($s); // 255
echo mb_strwidth($s); // 255

So indeed mb_strimwidth() may not be the issue in your specific case, but it would if you were storing CJK half-width chars: consider for example katakana Wo, half and full width ヲヲ: strlen=6, mb_strlen=2, mb_strwidth=3.

Normally I create my PostgreSQL databases with template1, which I assumed uses UTF-8 as encoding

That would be my expectation too, and indeed it seems to be the case on my dev box -

postgres=# \l template1
                                                 List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    | ICU Locale | Locale Provider |   Access privileges   
-----------+----------+----------+-------------+-------------+------------+-----------------+-----------------------
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | en-US      | icu             | =c/postgres          +
           |          |          |             |             |            |                 | postgres=CTc/postgres

on the affected DB and I get "SQL ASCII" as the result there

Well if you can only store 255 bytes instead of 255 UTF-8 chars, that would definitely explain things ;-)
And you'll definitely want to dump, recreate and reload your DB to switch the charset to UTF-8.

mpeterma

mpeterma

2023-04-12 03:47

reporter   ~0067638

@dregad: I just wanted to check back in for a second. After migrating my database from ASCII encoding to UTF-8 encoding, the problem I reported no longer occurs for me. For me, the reason was that I assumed the same defaults when switching from different operating systems or did not check them sufficiently. On NetBSD the PostgreSQL database must be created with encoding "UTF8", collate "C" and locale "de_DE.UTF-8". ICU is not enabled in the default version of pkgsrc. Thanks again for taking care of my problem. Furthermore, if the problem you noted with mb_strimwidth is problematic for special use cases, feel free to keep this ticket open for that. Otherwise, I would also agree to close it now. Best regards, Matthias

Translated with www.DeepL.com/Translator (free version)

dregad

dregad

2023-04-12 13:00

developer   ~0067643

Thanks for the feedback, and glad to hear you were able to fix the problem by converting the DB to UTF-8 character set.

I will open a separate issue to track the mb_strimwidth() issue.

dregad

dregad

2023-04-12 13:27

developer   ~0067644

See 0032385 for the mb_strimwidth() fix follow-up