Dynamically prerender pages for bots and crawlers, with Lambda@Edge, S3 and CloudFront. No more need for isomorphic/server-side rendering!
Dynamic Rendering is, as the name implies, about rendering a page based on a dynamic condition. What we want to do is solve the very common problem of modern Progressive Web Apps not working very well with some older crawlers that are unfortunately still in use across the web, such as on those used by Facebook and LinkedIn. Our condition is therefore ”if a crawler or bot sees the page, give it prerendered markup”.
Even Google thinks this is a pretty good approach for SEO. Refer to their documents Understand the JavaScript SEO basics and Implement dynamic rendering for further reading.
For this to work, we need to intercept the browser’s request BEFORE it actually hits the server to see whether the visitor is a bot or not. This repo assumes Amazon Web Services (AWS) for the entire solution. For the edge function capability, this responsibility could probably be doled out to Cloudflare Workers or similar as well.
The individual services used will be:
public
folder if you need something to experiment with.All of this magic can be fairly tedious to work with from scratch, especially if you are not accustomed to AWS. Combined with the slow deployment times, it’s not really the most user-friendly mini-project I’ve done. I hope I spare some of you out there a bit of wasted effort!
The below helps with the overall steps for setting everything up, with some steps automated and some manual. Since my last version, Serverless Framework has added more support for Lambda@Edge functions, but still not enough to make me capable of deploying all the functions and setting up Cloudfront the way it's supposed to be. Note that I am not saying it's not possible, but you won't find it working like that in this repo. You should be able to automate everything (minus function deployment) through Terraform if that's your poison of choice.
An improvement on this prerenderer would be to include render-caching capabilities so you need to do less processing.
Use whatever you want!
A basic, routed React application is available in the demo-site-s3
folder if you need something to experiment with. The routes are /thisview
and /thatview
.
Make sure you have an AWS account and that you're logged in through the terminal/environment.
deploy-demo-site.sh
and set a unique bucket namedeploy-demo-site.sh
https://uxj5ky4iw8.execute-api.eu-north-1.amazonaws.com/dev/prerender
https://uxj5ky4iw8.execute-api.eu-north-1.amazonaws.com/dev/prerender?url=http://d13x2tqlfdnb32.cloudfront.net/
PRERENDERER_ENDPOINT
constant in src/edge/originResponse.js
(should be line 3)Make sure you have an AWS account and that you're logged in through the terminal/environment.
index.html
snippets/bucket-policy.json
)snippets/cors.xml
)yarn
or npm install
sls deploy
to deploy the package with Serverlesshttps://uxj5ky4iw8.execute-api.eu-north-1.amazonaws.com/dev/prerender
https://uxj5ky4iw8.execute-api.eu-north-1.amazonaws.com/dev/prerender?url=http://d13x2tqlfdnb32.cloudfront.net/
PRERENDERER_ENDPOINT
constant in src/edge/originResponse.js
(should be line 3)http://prerender-demo.s3-website.eu-north-1.amazonaws.com
)/index.html
This is going to deploy for 5-15 minutes, so continue with the rest below.
This is highly recommended, and will save you lots of time compared to doing it manually.
npm run deploy:edge
I am overreaching a bit on what to create, but at least the following works for me.
AWSLambdaBasicExecutionRole
and attach itsnippets/lambda-log-policy.json
, save itsnippets/lambda-iam-trust-policy.json
, save itarn:aws:lambda:us-east-1:487572315234:function:PrerenderRequest:12
):12
or whatever the number might be - this corresponds to the version number, if you don't see it click the Qualifiers/Version box and switch to the version with the highest numberAccess-Control-Request-Headers
, Access-Control-Request-Method
, Origin
, x-prerender-uri
, x-resolved-user-agent
, x-should-prerender
:12
(or whatever number) bitServices that should be able to give you an indication of functionality:
If you use Insomnia or Postman, you can try GETting your Cloudfront distribution with a user-agent
header like googlebot
and it should respond with rendered HTML. Removing the header should in turn start returning the basic non-rendered HTML again.
Check the default Cloudwatch logs at console.aws.amazon.com/cloudwatch/. Edge functions are currently only available in the us-east-1
region, so look for those logs in that region. Logs may also show up in whatever region the CDN thinks you are in, so in case you actually don't find any logs, look in regions close to you.
When you test Facebook or similar, the same goes for those: These are most likely going to be run from one of the North American regions.
Run the teardown.sh
script. Also read the below from Serverless Framework's blog post on Lambda@Edge support:
**A note about removals**
When you’re done with testing you might want to remove the service via serverless remove. Note that the removal also takes a little bit longer and won’t remove your Lambda@Edge functions automatically. The reason is that AWS has to cleanup your functions replicas which can take a couple of hours. Removing the Lambda functions too early would result in an error.
The solution for this problem right now is to manually remove the Lambda@Edge functions via the AWS console after a couple of hours. You might want to automate this process with a script which issues AWS SDK calls to streamline this cleanup process.
Long story short: You will need to remove Cloudfront and Lambda@Edge functions manually.
Many steps, but that should about do it to add dynamic rendering!