Support or be clear about lack of support for Lambda -> SNS -> SQS -> Lambda

[mikeparker]

I've spent many hours trying to get Lambda -> SNS -> SQS -> Lambda to work in xray, and all my traces end with SNS, never going through to SQS and the next lambda.

My code is not very complicated.

	sess := session.Must(session.NewSession())
	client := sns.New(sess)
	xray.AWS(client.Client)

// do some work

	_, err = client.PublishWithContext(ctx, msg)

AWS documentation says that SNS and SQS should automatically pass on the tracing header to the next lambda, and I can't see anything on my SQS queues configuration to enable tracing, it should just work, right?

I eventually found this comment on the NodeJS library:
aws/aws-xray-sdk-node#208 (comment)

This is a known issue with propagating trace context through an SQS queue. It is not currently possible to connect a trace passed into an SQS queue to the Lambda consuming it on the other end. This is because although the trace header is propagated in the SQS message, you cannot set it in the Lambda because segments in Lambda are immutable.
Enabling this functionality is a feature that is in progress and one we hope to release for all our SDKs. Stay tuned and I will update this issue as we make progress in the Node SDK.

Can you confirm if the golang library has the same issue (due to an underlying issue with Lambda perhaps?) and there is no way of it working (and I am wasting my time trying)?

The only other thing I found was this https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html which shows some code to manually recreate a segment (Excuse the Java code)

    // Retrieve the trace header from the AWSTraceHeader message system attribute
    String traceHeaderStr = message.getAttributes().get("AWSTraceHeader");
    if (traceHeaderStr != null) {
        TraceHeader traceHeader = TraceHeader.fromString(traceHeaderStr);

        // Recover the trace context from the trace header
        Segment segment = AWSXRay.getCurrentSegment();
        segment.setTraceId(traceHeader.getRootTraceId());
        segment.setParentId(traceHeader.getParentId());
        segment.setSampled(traceHeader.getSampled().equals(TraceHeader.SampleDecision.SAMPLED));
    }

I assume this isn't possible in Lambda for the reasons above (lambda segments being immutable)?

[srprash]

Hi @mikeparker
Yes, right now all the Lambda platforms do not support propagation of trace header from SQS to a lambda function. We are hard at work on it.

Can you confirm if the golang library has the same issue (due to an underlying issue with Lambda perhaps?) and there is no way of it working (and I am wasting my time trying)?

Yes, a lambda function with a golang runtime has the same limitation.

I assume this isn't possible in Lambda for the reasons above (lambda segments being immutable)?

Yes, you are right about this as well. And I agree with you that we should document this limitation.

We will be posting updates once this feature is supported. Please stay tuned.

Thanks

[mikeparker]

Thanks for the quick reply @srprash. Good to know i'm not going crazy!

I know its hard to talk about release dates but we need to make a call about whether we're going to roll our own tracing system over the next few months, is this something that might arrive this year? Are we talking months or years roughly?

[srprash]

Hi @mikeparker
Since we cannot openly talk about the release date, I have asked the AWS support to reach out to you with the information. Thanks for your patience.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[xrn]

Hi @srprash

Any update on this topic?

[srprash]

Hi @xrn
The X-Ray team is working hard to get this support rolled out. We will update this issue once the feature is available.
Thanks for being patient.

[dfens1]

thanks, @mikeparker - you saved me hours of wasted efforts!!

[Psykar]

So poking at this a bit, I believe it would be possible to hack this by creating a custom emitter, and using https://docs.aws.amazon.com/sdk-for-go/api/service/xray/#XRay.PutTraceSegments inside the Emit() method on the custom Emitter type.

Note I haven't actually tested this, but seems a plausible workaround to skip the lambda's xray daemon.

type Emitter struct{}

func (e *Emitter) Emit(x *xray.Segment) {
	fmt.Println("segment: ", x)
        // TODO - https://docs.aws.amazon.com/sdk-for-go/api/service/xray/#XRay.PutTraceSegments
}

func (e *Emitter) RefreshEmitterWithAddress(*net.UDPAddr) {}

func stuff() {
	xray.Configure(xray.Config{
		Emitter: &Emitter{},
	})
	ctx, seg := xray.NewSegmentFromHeader(context.TODO(), "somename", nil, &header.Header{
		TraceID:  "traceid",
		ParentID: "parentid",
	})
	defer seg.Close(nil)

	_, subset := xray.BeginSubsegment(ctx, "subseg")
	defer subset.Close(nil)
}

Besides being an awful hack, is there anything fundamentally wrong with this approach?

[willarmiros]

@Psykar that workaround should work, as long as you have X-Ray permissions on your lambda. Just also make sure to attach the correct trace ID, parent ID, and sample decision from the _X_AMZN_TRACE_ID environment variable in order to parent your hacky segment correctly.

[mariotoffia]

Hi @dfens1 is there any progress on this issue - I'm using the Go SDK v2 and would very much like to have the ability to do this or at least Lambda->SQS->Lambda.

(and if possible do Lambda->SQS->Lambda->IoT Core Device Shadow->Kinesis Firehose->Lambda->S3->Lambda 😊))

Cheers,
Mario :)