Archive and search your tweets and liked tweets using AWS Lambda, DynamoDB and Elasticsearch.
Note: this project is primarily being used a test bed for figuring out best practices with AWS Lambda
Make sure you have the following installed before you proceed
- AWS CLI - Install AWS CLI configured with Administrator permission
- SAM CLI - Install the SAM CLI
- AWS Amplify - Install AWS Amplify
- Python 3
- Docker - Install Docker community edition
Twitter API key setup
Set up a Twitter Developer account. Once you are signed up, create an app.
Add the credentials to SSM Parameter Store:
aws ssm put-parameter --name /tweeter/twitter/consumer_key --value <your consumer key value> --type SecureString --overwrite aws ssm put-parameter --name /tweeter/twitter/consumer_secret --value <your consumer secret value> --type SecureString --overwrite aws ssm put-parameter --name /tweeter/twitter/access_token --value <your access token value> --type SecureString --overwrite aws ssm put-parameter --name /tweeter/twitter/access_token_secret --value <your access token secret value> --type SecureString --overwrite
Elastic Cloud setup
Elastic Cloud is not free
Set up an account with Elastic Cloud. Create a deployment and then an Elastic App Search engine. Retrieve the private key from the Credentials section.
Add the private key to SSM.
aws ssm put-parameter --name /tweeter/es/private_key --value <your password> --type SecureString --overwrite
Also, update the configuration in
ui/src/config/engine.json, specifically the values of
Deploy the backend
Build and deploy your application for the first time by running the following commands in your shell:
make build make deploy.guided
The first command will build the source of your application within a Docker container. The second command will package and deploy your application to AWS. Guided deploy means SAM CLI will ask you about the name of your deployment/stack, AWS Region, and whether you want to save your choices, so that you can use
make deploy next time.
Deploy the frontend
ui directory, deploy the application with AWS Amplify.
Use the SAM CLI to build and test locally
Whenever you change your application code, you'll have to run build command:
The SAM CLI installs dependencies defined in
poller/requirements.txt, creates a deployment package, and saves it in the
Test a single function by invoking it directly with a test event:
make invoke.poller make invoke.indexer
Working with localstack
Validate DynamoDB was populated using Localstack:
docker-compose up -d make invoke.poller aws --endpoint-url=http://localhost:4566 dynamodb scan --table-name TweetsTable --max-items 10
An event is a JSON document that represents the input that the function receives from the event source. Test events are included in the
eventsfolder in this project.
Fetch, tail, and filter Lambda function logs
To simplify troubleshooting, SAM CLI has a command called
sam logs lets you fetch logs generated by your deployed Lambda function from the command line. In addition to printing the logs on the terminal, this command has several nifty features to help you quickly find the bug.
NOTE: This command works for all AWS Lambda functions; not just the ones you deploy using SAM.
sam logs -n TweeterPoller --stack-name tweeter --tail
You can find more information and examples about filtering Lambda function logs in the SAM CLI Documentation.
CI/CD with GitHub
Create a user
github with the following policy: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-permissions.html.
Create secrets with AWS.
Follow the AWS Amplify console to configure the frontend.
samconfig.tomlthe first time you run the poller to get all tweets.
- Twitter apparently only allows access to a user's most recent 3,240 tweets with this method.
- This probably isn't the best use case for Lambda, which has a timeout of 15 minutes. Ideally we should be calling Fargate for long running task. That said, in practice, because of the limitations of Twitter's API, getting all the available tweets is well within that limit.
Tracer utility patches known libraries, and trace the execution of this sample code including the response and exceptions as tracing metadata - You can visualize them in AWS X-Ray.
Logger utility creates an opinionated application Logger with structured logging as the output, dynamically samples 10% of your logs in DEBUG mode for concurrent invocations, log incoming events as your function is invoked, and injects key information from Lambda context object into your Logger - You can visualize them in Amazon CloudWatch Logs.
Metrics utility captures cold start metric of your Lambda invocation, and could add additional metrics to help you understand your application KPIs - You can visualize them in Amazon CloudWatch.