Skip to content

feat: Add Sync and Async AWS Lambda Ingest#72

Merged
jshlbrd merged 6 commits intomainfrom
jshlbrd/aws-lambda-ingest
Jan 19, 2023
Merged

feat: Add Sync and Async AWS Lambda Ingest#72
jshlbrd merged 6 commits intomainfrom
jshlbrd/aws-lambda-ingest

Conversation

@jshlbrd
Copy link
Contributor

@jshlbrd jshlbrd commented Jan 17, 2023

Description

  • Adds ingest support for sync invocation in the AWS Lambda app
  • Adds ingest support for async invocation in the AWS Lambda app
  • Moved code out of main.go and into individual Go files -- no changes, just an attempt to organize the code
  • Updated AWS examples to support microservices and pipelines
    • The pipeline example has two new components: async Lambda ingest and a sync Lambda deployed as an enrichment microservice

Motivation and Context

Synchronous invocation partially addresses #54 and turns Substation into a microservice. The primary use case for this is integrating Substation processors with external applications, but this is also compatible with the AWS Lambda processor (which could be used to allow Substation pipelines to invoke centralized enrichment Lambda).

Asynchronous invocation was an easy add (copy/paste from sync). Both have some limitations, which are documented in the code and copied here:

  • Both require the input payload to be an object (JSON)
  • Sync invocation must return a single object (JSON)
  • Sync invocation must be configured to use the gRPC sink with results sent to localhost:50051

This shouldn't require any new build support (e.g., Terraform, configurations), but if we discover that it does then we can add it as a chore PR later.

How Has This Been Tested?

Integration tested using the new microservice and refactored pipeline AWS examples.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@jshlbrd jshlbrd marked this pull request as ready for review January 18, 2023 17:59
@jshlbrd jshlbrd requested a review from a team as a code owner January 18, 2023 17:59
@jshlbrd jshlbrd merged commit 141fdf5 into main Jan 19, 2023
@jshlbrd jshlbrd deleted the jshlbrd/aws-lambda-ingest branch January 19, 2023 01:17
@jshlbrd jshlbrd linked an issue Jan 19, 2023 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Request-Reply Microservices / Remote Procedure Calls

2 participants