Repair parsing error due to commented-out field token

[gittaca]

When a line starts with # but continues with one of the valid fields / tokens, this backtrace is produced:

Error: Test failed: 'Commented-out tokens get parsed correctly'
* 'names' attribute [2] must be the same length as the vector [0]
Backtrace:
 1. testthat::expect_true(...)
 5. robotstxt::parse_robotstxt(rtxt_ct)
 6. robotstxt::rt_get_fields(txt, "allow") …robotstxtRparse_robotstxt.R:7:2
 7. base::lapply(...) …robotstxtRrt_get_fields.R:38:2
 8. robotstxt:::FUN(X[[i]], ...)

After removing the # in this demo .txt, the test case will fail with nrow(parse_robotstxt(rtxt_ct)$permissions) == 1 isn't true. which I take that the error is avoided.

Maybe the error results from somewhere in rt_get_fields.R? Maybe because one of the reg-exes detects the valid token ([2]), but another detects the line as a comment ([0]), thus clashing at names(fields) <- c("field", "value")?

I've tinkered with the various reg-exes there for about an hour, but found no solution. Will instead remove the problematic file from my analysis to continue with that for now.

Please feel free to take over the PR. I hope the test is a useful start to find and fix the problem.

[petermeissner]

Thanks, can you open an issue for that please.

[petermeissner]

@gittaca also, can you give a reproducible example, please.

[gittaca]

I included a inst/robotstxts/robots_commented_token.txt in this PR ;-)