aws logs filter-log-events: Error during pagination: The same next token was received twice

[dflantz]

Hi! Here are the details:

$ aws --version
aws-cli/1.14.50 Python/3.6.4 Darwin/16.7.0 botocore/1.9.3

$ aws logs filter-log-events --log-group-name=<log-group-name>   \
--start-time=1518998400000 --end-time=1519084800000   \
--filter-pattern='<filter-pattern>'  --query "events" > logs.json

Error during pagination: The same next token was received twice: {'nextToken': '<token>'}

I believe this may be related to #1485.

[joguSD]

Is this an intermittent issue or does this happen all of the time?
If it's broken fairly consistently we may have to remove this paginator.

[starkshaw]

My customer has the same issue. Let me ask him to try to run it multiple times.

[dasmowenator]

It happens to me quite regularly when using the AWS CLI to search for CloudWatch Logs, but it's non-deterministic. When I get this error from my scripts, I just re-run them and it's a 50-50 chance whether it'll happen again or not.

[dasmowenator]

I actually do think it's reproducible, but I'm not 100% sure how. Generally, if I run a "aws logs filter-log-events" request against a very large log group (like an API Gateway execution log group) which takes a long time, if I try to run any other requests against the same log group at the same time, I get the error. Or if I Ctrl+C my original request while it's running and then retry the command I get the same error.

[dasmowenator]

So I responded... and even included steps to reproduce the issue... why is it still marked as "closing soon if no response"?

[tyvich]

Also have recently encountered this issue with a log group containing large items, but not necessarily retrieving a high volume of items (~5000). Was able to remedy by providing pagination config parameter, PageSize, of seemingly any arbitrary value.

Was not able to reproduce by iterating through filter_log_events without paginator and looking for duplicate tokens.

[JustinTArthur]

Seeing this on boto3/botocore as well on a high volume log retrieval. Has something changed recently on the underlying APIs?

Traceback (most recent call last):
  File "/Users/jarthur/Repos/Lifesize.misc/jarthur/Reports/process_cloudwatch_vpss.py", line 51, in <module>
    for page in response_iterator:
  File "/Users/jarthur/.virtualenvs/misc/lib/python3.5/site-packages/botocore/paginate.py", line 301, in __iter__
    raise PaginationError(message=message)
botocore.exceptions.PaginationError: Error during pagination: The same next token was received twice: {'nextToken': 'Bxkq6kVGFtq2y_Moigeqsd0gU4kG897U2M9-6GKlGbZoYxBDfCbY…

[JustinTArthur]

I just got word from AWS that they are aware of the issue and working on a fix. @joguSD it seems this issue is erroneously labeled as awaiting a response.

[dmuzzy]

Been getting this error for about a week now, and no pagination settings seem to get around it. On Windows 10, I just updated my awscli install from 1.14.66 to 1.15.6, but that didn't help.

@JustinTArthur Any word from AWS on this?

[JustinTArthur]

I queried for workarounds, but haven't heard back. As soon as I hear about the fix or a workaround, I'll follow up here.

[dmuzzy]

Cool, thanks.

[margeson]

Issue is happening in version 1.15.13.

AWS CLI was installed using pip

Seems to choke when you are pulling logs greater than a month

$ aws logs filter-log-events --log-group-name flowlogs --log-stream-names eni-11111-all --filter-pattern "[version="2", account_id, interface_id, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, action="ACCEPT", log_status="OK"]" --start-time 1522782087000

Error during pagination: The same next token was received twice: {u'nextToken': u'zpUlahi1iz0r54spQh9r4Squ0R9zaukaZh7g5OILz5dLsJokO6t1ZmDdPShgBRpvAD3lTwSePeDHB3cJs6kiIgBY0ueYP6ydqnHIzvRNdN8joRobMpKIqc_sYf425QfL8qNuDlUxFBq_Zmt9eIEmCPzFJMIWtfZYI2F5jEgyx0Pc85ut9luVlCB6-iSKyV9CWOzNeltbrkn-T7hwO-BhKC5jNPLsKgbwPmgE02Qa-ozdgIvx2mo1ccGoXWodcYEP'}
$ aws --version aws-cli/1.15.13 Python/2.7.12 Linux/4.10.0-38-generic botocore/1.10.13

[JustinTArthur]

While I haven't heard official word back from AWS yet, this started working again for me in the us-west-1 region as of 48 hours ago using botocore stand-alone. For those of you experiencing it, let me know if it's fixed for you, whether you're using aws-cli or a boto lib.

[dasmowenator]

I'm still experiencing this error.