Problem to get header content!

[jungjin09]

Hi,

Please check the attached file.
The separator character is tab.
The third column header returns the wrong content.
Thanks you.

B.TXT

[jungjin09]

Hi,
I realized the problem lies with the unicode characters that are at the beginning or end of each cell. will cause field_start and field_length calculations to be incorrect.
The temporary way I came up with is to use a variable that defines the boundary:

const BOOL checkboundary(char x) {
return x == '\t' || x == ',';
};
const auto& isbeginend = data_pos == 0 || (data_pos > 1 &&
((!checkboundary(this->data_ptr->data[data_pos]) && checkboundary(this->data_ptr->data[data_pos - 1])) ||
(!checkboundary(this->data_ptr->data[data_pos]) && data_pos < size - 1 && checkboundary(this->data_ptr->data[data_pos + 1]))));

From there recalculate the flag:

const auto& flag = compound_parse_flag(in[data_pos], isbeginend);

It may work in some cases but not completely.
B.TXT

[jungjin09]

Hi @txemaotero,
I have tried the latest version (single_header) but the result is still the same.
Did I miss anything? because it's been a while.

If I do not specify the delimiter then suggest delimiter gets wrong recognition. It does not recognize the separator character as Tab.

When I specify the tab separator character then I get the following result.
Before:

After:

Going into more detail it seems the problem lies in IBasicCSVParser::parse()

it is the functions

compound_parse_flag(char
parse_flag(char
ws_flag(char
IBasicCSVParser::parse_field()

Related code:

default: // Quote (currently not quote escaped)
data_pos++; //if there is a delimiter before it then it should not be incremented
if (field_start == UNINITIALIZED_FIELD && data_pos < in.size() && !ws_flag(in[data_pos]))
field_start = (int)(data_pos - current_row_start());
data_pos += 1;

However the code works fine with ansi.
Thanks.

B (1).TXT

[txemaotero]

Hi!
First, sorry for the noise 🙏 I was sure the code I pushed worked but when trying to test it now it wasn't 😓 I wanted to attach originally a link to Compiler Explorer showing the error but, although some months ago I was able to achieve it, for some reason now I wasn't (I think the compilation in arm arch was taking longer that permited).

Any way, I've introduced a new constant (CHAR_OFFSET) that wasn't being defined properly. I've pushed a fix right now and my local tests are ok now. Indeed I used the file you provided. I've a tests setup that compiles and runs this simple code:

int main()
{
    using namespace csv;
    CSVFormat format;
    format.delimiter('\t');
    auto rows = parse(R"(Mã SP	Tên	ẻm lạ
XB01	Xa bông LifeBoy	4200
S001	Đinh tiến hự	4350)", format);

    for (auto& row: rows) {
        for (auto& field: row) {
            std::cout << field.get<>() << ", ";
        }
        std::cout << "\n";
    }
    return 0;
}

Using the version in master and the version in my PR both in linux x64 and arm64. Here is the output:

Running master x64:
XB01, Xa bông LifeBoy, 4200, 
S001, Đinh tiến hự, 4350, 

Running master arm64:
XB01, Xa bông LifeBoy, 4200, 
S001, inh tiến hư, 4350, 

Running txema x64:
XB01, Xa bông LifeBoy, 4200, 
S001, Đinh tiến hự, 4350, 

Running txema arm64:
XB01, Xa bông LifeBoy, 4200, 
S001, Đinh tiến hự, 4350, 

As you can see, in x64 the differences you pointed out are vissible but they disappear using the version PR. I think it is more clear using emojis 😅:

Running master x64:
Hello😀, world😍, 

Running master arm64:
Hello�, world�, 

Running txema x64:
Hello😀, world😍, 

Running txema arm64:
Hello😀, world😍, 

I hope this clarifies your concerns and again sorry for wasting your time with the previous wrong version 🙏