Adding a CI/CD Pipeline

2019-02-18 Technical and Programming

To reduce the chance of a human error affecting the a production site (especially in a team environment) it is good practice to implement a CI/CD pipeline. This post covers the implementation of a CI/CD pipeline for the AlphaGeek site. In future I will provide a more generic post about CI/CD pipelines.

# Pre-requisites

This article will be based on the blog described in Hexo, AWS and Serverless Framework,

Securing A Test Environment Using AWS WAF, Securing S3 Bucket from Direct Access and Implementing Unit Testing; some of the concepts can be applied to other uses, but this post will focus on the

specific AlphaGeek use case.

# Select a CI/CD Pipeline Provider

There are a large number of CI/CD Pipeline providers. Some of them offer self-hosted solutions, others charge for even the most basic account, some are free for open-source projects, and some provide limited accounts for free.

For my purposes I selected SemaphoreCI as it integrates with GitHub, is free (with significant usage limitations) and appears to provide a high level of configuration.

# Design Your Pipeline

As the implementation of a CI/CD pipeline is to improve the reliability of my blog I added a number of new packages to my blog’s requirements as part of the CI/CD pipeline.

If any task in the pipeline fails all subsequent actions should not occur. The final pipeline design will operate as follows:

# Pre-commit Git Hook

A pre-commit git hook will prevent any commits of invalid code.

  • Run unit tests;
  • Run markdown linting;
  • Clean the site build files;
  • Rebuild the site;
  • Apply versioning to the asset files;
  • Scan the generated files for broken links;
  • Scan the generated files for non-W3C compliant HTML;
  • Scan the generated files for accessibility standards compliance but ignore the results.

# CI/CD Pipeline

The following CI/CD pipeline will be triggered when a commit to the develop branch is pushed to GitHub.

  • Re-run the pre-commit tasks in the CI/CD environment;
  • Deploy content changes to the development site;
  • Scan the deployed files for broken links.

The following CI/CD pipeline will be triggered when a commit to the master branch is pushed to GitHub.

  • Re-run the pre-commit tasks in the CI/CD environment;
  • Deploy content changes to the production site;
  • Scan the deployed files for broken links.

After all tests have passed an option to deploy infrastructure and code changes will be available though the chosen CI/CD SaaS provider.

# Pipeline Implementation

# Sign-up to SemaphoreCI

The first task was to sign-up to SemaphoreCI. This was as simple as clicking on the large Sign up with GitHub button and selecting the repository I wanted to integrate.

# Set-up the New Test Dependencies

To implement the tests several new dependencies have been added to the blog.

# markdownlint-cli

To ensure the markdown for all the posts is formatted consistently markdownlint-cli was installed.

1
npm i --save-dev markdownlint-cli

A custom configuration was created for this dependency so it operates how I want it to. This configuration enforces the top level heading as level 2 (level 1 headings are used automatically for the post title in my chosen template); the maximum line length was disabled because it is not compatible with Hexo’s default Markdwon interpreter; and I removed the ? character from the heading validation.

1
2
3
4
5
6
7
8
9
{
  "MD002": {
    "level": 2
  },
  "MD013": false,
  "MD026": {
    "punctuation": ".,;:!"
  }
}

# node-w3c-validator

To ensure there is no invalid HTML on the site I implemented node-w3c-validator. This script has a dependency on Java, so that may need to be installed as well.

1
npm i --save-dev node-w3c-validator

# pa11y-ci

Pa11y-CI is a wrapper for Pa11y to make it easier to integrate in a CI/CD pipeline.

Pa11y scans an HTML file for any accessibility issues. As my blog has a large number of existing issues I have configured the scripts that run this to allow it to fail.

1
npm i --save-dev pa11y-ci

# Add New Scripts to NPM

To simplify running the existing NPM commands and the new dependencies I added a number of elements to the scripts section of the package.json file, modified some of the definitions and re-ordered them to make more sense to me.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
"scripts": {
  "jest": "jest",
  "mdlint-drafts": "markdownlint --config .mdconfig ./source/_drafts",
  "mdlint": "markdownlint --config .mdconfig ./source/_posts",
  "precheck": "npm run jest && npm run mdlint",
  "clean": "hexo clean",
  "build": "hexo generate",
  "cleanbuild": "npm run clean && npm run build",
  "linkcheck": "blcl --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ public",
  "htmlcheck": "node-w3c-validator -v -s -i public/",
  "a11ycheck": "pa11y-ci public/*.html public/*/*.html public/*/*/*.html public/*/*/*/*.html public/*/*/*/*/*.html || true",
  "validate": "npm run linkcheck && npm run htmlcheck && npm run a11ycheck",
  "buildtest-local": "npm run precheck && npm run cleanbuild && npm run validate",
  "precommit": "npm run buildtest-local"
},

# Default theme issues

If you are running the default Hexo theme (landscape), it is not W3C compliant, so you will need to change the htmlcheck script in your package.json to be node-w3c-validator -v -s -i public/ || true. This will display the output when you run the precommit hook, but will not enforce the HTML validation.

# Implement the Git Pre-Commit Hook

Implementing a Git pre-commit hook is simply a matter of creating a file at .git/pre-comit and populating it with a valid shell script that completes with an error code of 0.

1
2
3
#!/bin/sh

npm run precommit

# Define the Tasks to Run on SemaphoreCI

The SemaphoreCI configuration is done in multiple phases to ensure it works and so a broken build isn’t accidentally deployed to production.

In the configuration file, tasks will be run sequentially, and if multiple jobs are defined within a task they can be run in parallel.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
version: v1.0
name: Hexo Serverless Build Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Prepare the build environment
  - name: Prepare
    task:
      jobs:
        # Make sure we've got the right Java version configured
        - name: Set Java Version
          commands:
            - change-java-version 8
        # Run NPM install, using Semaphore's cache where possible
        - name: NPM Install
          commands:
            # Update NPM because it's so old
            - cache restore npm
            - npm i -g npm
            - cache store npm .nvm/versions/node/v8.11.3/lib/node_modules/npm
            - checkout
            # Reuse dependencies from cache and avoid installing them from scratch:
            - cache restore node-modules-$(checksum package-lock.json)
            - npm ci
            - cache store node-modules-$(checksum package-lock.json) node_modules
  # Run the validation routines that don't require a build
  - name: Validate
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Run the jest test suite
        - name: Run Jest Tests
          commands:
            - npm run jest
        # Run the Markdown linter
        - name: MD Lint
          commands:
            - npm run mdlint
  # Build the deployment files
  - name: Build
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Ensure we have a clean build directory, generate the files and add asset versioning
        - name: Build Site
          commands:
            - npm run clean
            - npm run build
            - cache store public-$(find source -type f -exec cat {} + | checksum) public
  # Run tests on the deployment files
  - name: Test Locally
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
          - cache restore public-$(find source -type f -exec cat {} + | checksum)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck
        # Check that the HTML is valid
        - name: Test W3C compatability
          commands:
            - npm run htmlcheck
        # Check if we meet a11y standards
        - name: Test Accessibility
          commands:
            - npm run a11ycheck

# Commit and Test

As a pre-commit hook has been added your next commit may be rejected until all existing errors have been fixed. In my case I had to fix errors in all of the markdown files for each of my previous posts.

At this point the CI/CD pipeline should successfully build and test the site.

# Add UAT and Production Scripts to NPM

To enable the automated deployment of content changes, and to provide functionality to deploy infrastructure and code changes, some additional scripts need to be defined in the package.json file. These scripts will allow for deployment and testing of both the UAT and Production environments.

1
2
3
4
5
6
7
8
9
10
11
12
"deploy-uat-infra": "npx serverless deploy -s dev",
"deploy-uat-site": "npx serverless s3deploy -s dev -v",
"deploy-uat": "npm run deploy-uat-infra && npm run deploy-uat-site",
"linkcheck-uat": "npx blc --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ --user-agent '**PASSWORD_DEFINED_IN_SERVERLESS_CONFIGURATION** Tester' http://dev.alphageek.com.au",
"test-uat": "npm run deploy-uat && npm run linkcheck-uat",
"buildtest-uat": "npm run buildtest-local && test-uat",
"deploy-prod-infra": "npx serverless deploy -s prod",
"deploy-prod-site": "npx serverless s3deploy -s prod -v",
"deploy-prod": "npm run deploy-prod-infra && npm run deploy-prod-site",
"linkcheck-prod": "npx blc --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ http://alphageek.com.au",
"test-prod": "npm run deploy-prod && npm run linkcheck-prod",
"buildtest-prod": "npm run buildtest-uat && test-prod"

# Remove Deployment Functionality from Serverless Framework

As the deployment process has been migrated to NPM commands the serverless.yml configuration file needs to the build and deployment functionality removed. Delete the following lines from the file:

1
2
3
4
5
6
7
8
9
scripts:
  hooks:
    # Run these commands when creating the deployment artifacts
    package:createDeploymentArtifacts: >
      hexo clean &&
      hexo generate
    # Run these commands after infrastructure changes have been completed
    deploy:finalize: >
      sls s3deploy -s ${self:custom.stage}

# Install SemaphoreCI CLI

To enable deployment to AWS SemaphoreCI will need to have access to our AWS credentials. Credentials and other secrets should never be stored in a code repository, so we will require a method to securely save the credentials on Semaphore. This can be done using the SemaphoreCI command line utility. You will also need to know your SemaphoreCI organization name and the SemaphoreCI API token (which can be found at the SemaphoreCI Account Page).

1
2
curl https://storage.googleapis.com/sem-cli-releases/get.sh | bash
sem connect **ORGANIZATION**.semaphoreci.com **API_TOKEN**

# Provide AWS Credentials to SemaphoreCI

To provide credentials and secrets to SemaphoreCI a file needs to be created. So this doesn’t accidentally get committed to the code repository we begin by adding to the .gitignore file.

1
2
# Semaphore secret files
.semaphore/secrets/*

It’s now safe to create a file with your AWS credentials in it. Create a new file at .semaphore/secrets/aws.yml with the following content (updated with your AWS details).

1
2
3
4
5
6
7
8
9
10
apiVersion: v1beta
kind: Secret
metadata:
  name: **YOUR_SITE_NAME**-aws
data:
  env_vars:
    - name: AWS_ACCESS_KEY_ID
      value: "**YOUR_ACCESS_KEY_ID**"
    - name: AWS_SECRET_ACCESS_KEY
      value: "**YOUR_SECRET_ACCESS_KEY**"

Then this file needs to be imported to SemaphoreCI.

1
sem create -f .semaphore/secrets/aws.yml

The value of these secrets will be embedded in the deployment configuration in the next section.

# Create UAT and Production Deployment Configuration

Everything is now ready to create the SemaphoreCI deployment scripts.

Start by creating a file to deploy the content changes to S3 for UAT at .semaphore/uat-content.yml.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek UAT Content Deployment Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Use Serverless to deploy to UAT
  - name: Publish to UAT
    task:
      # Import the secret environment variables
      secrets:
        - name: alphageek-aws
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
          - cache restore public-$(find source -type f -exec cat {} + | checksum)
      jobs:
        - name: Deploy Content
          commands:
            - npm run deploy-uat-site
  # Run tests on the UAT site
  - name: Test UAT
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck-uat

Create a similar file for deploying content changes to S3 for Production at .semaphore/prod-content.yml.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek Production Content Deployment Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Use Serverless to deploy to Production
  - name: Publish to Prod
    task:
      # Import the secret environment variables
      secrets:
        - name: alphageek-aws
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
          - cache restore public-$(find source -type f -exec cat {} + | checksum)
      jobs:
        - name: Deploy Content
          commands:
            - npm run deploy-prod-site
  # Run tests on the Production site
  - name: Test Production
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck-prod

Now we create files for both UAT and Production infrastructure and Lambda function changes at .semaphore/uat-infra.yml and .semaphore/prod-infra.yml respectively.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek UAT Infrastructure Deployment Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Use serverless to deploy to UAT
  - name: Publish to UAT
    task:
      # Import the secret environment variables
      secrets:
        - name: alphageek-aws
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        - name: Deploy Infrastructure
          commands:
            - npm run deploy-uat-infra
            - cache store serverless-$SEMAPHORE_GIT_BRANCH .serverless
  # Run tests on the UAT site
  - name: Test UAT
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck-uat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek Production Infrastructure Deployment Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Use serverless to deploy to Production
  - name: Publish to Production
    task:
      # Import the secret environment variables
      secrets:
        - name: alphageek-aws
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        - name: Deploy Infrastructure
          commands:
            - npm run deploy-prod-infra
            - cache store serverless-$SEMAPHORE_GIT_BRANCH .serverless
  # Run tests on the UAT site
  - name: Test Production
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck-prod

# Add UAT and Production Deployment Configuration to Semaphore

Now the deployment processes have been defined we need to add triggeres for them to the primary SemaphoreCI configuration file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
promotions:
  - name: Deploy Content to UAT
    pipeline_file: uat-content.yml
    auto_promote_on:
      - result: passed
        branch:
          - ^develop$
  - name: Deploy Infra to UAT
    pipeline_file: uat-infra.yml
  - name: Deploy Content to Production
    pipeline_file: prod-content.yml
    auto_promote_on:
      - result: passed
        branch:
          - ^master$
  - name: Deploy Infra to Production
    pipeline_file: prod-infra.yml

# Add New WAF Rules and Implement

Because the link checking tool we’re using doesn’t support a custom authentication header we need to enable another method to gain access. For this we will use a custom user-agent string. This is defined in config/resources.yml.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
CustomUserAgentHeader:
  Type: AWS::WAF::ByteMatchSet
  Properties:
    ByteMatchTuples:
      -
        FieldToMatch:
          Type: HEADER
          Data: User-Agent
        TargetString:
          Fn::Join:
            - " "
            - - ${self:custom.security.passwords.development}
              - "Tester"
        TextTransformation: NONE
        PositionalConstraint: EXACTLY
    Name:
      Fn::Join:
        - "_"
        - - ${self:custom.domain.domainname}
          - UserAgent
          - Header
CustomUserAgentHeaderRule:
  Type: AWS::WAF::Rule
  Properties:
    Name:
      Fn::Join:
        - "_"
        - - ${self:custom.domain.domainname}
          - UserAgent
          - Header
          - Rule
    MetricName:
      Fn::Join:
        - ""
        - - ${self:custom.domain.domain}
          - ${self:custom.stage}
          - UserrAgent
          - Header
          - Rule
    Predicates:
      -
        DataId:
          Ref: CustomUserAgentHeader
        Negated: false
        Type: ByteMatch

Then add a reference to it in config/other.yml.

1
2
3
4
5
6
-
  Action:
    Type: ALLOW
  Priority: 2
  RuleId:
    Ref: CustomUserAgentHeaderRule

# Deploy New Infrastructure

We need to manually deploy the new infrastructure before we can push the code to GitHub as some of the functionality in the CI/CD pipeline will fail with the current configuration.

1
npm run deploy-uat-infra

# How to Publish New Content

Publishing new content is now as easy as committing the changes and pushing the develop branch to GitHub and waiting for the deployment to the development site to complete. Once you’ve completed any user acceptance testing (UAT) you can merge the develop branch into master and push that to GitHub. Once all testing has completed the content will automatically be published to your production site.

# How to Deploy New Infrastructure and Lambda Functions

Deploying new infrastructure is similar to the process for new content, but once the tests and build have completed on SemaphoreCI you will need to press a button to deploy. Just follow these simple steps:

  • Login to SemaphoreCI
  • Locate the build you wish to deploy the infrastructure from
  • Open the build
  • Click the Promote button under the deployment you wish to run

# The Final Configuration Files

serverless.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
# The name of your project
service: **project**

# Plugins for additional Serverless functionality
plugins:
  - serverless-s3-deploy
  - serverless-plugin-scripts

# Configuration for AWS
provider:
  name: aws
  runtime: nodejs8.10
  profile: serverless
  # Some future functionality requires us to use us-east-1 at this time
  region: us-east-1

  # This enables us to use the default stage definition, but override it from the command line
  stage: ${opt:stage, self:provider.stage}
  # This enables us to prepend the stage name for non-production environments
  domain:
    fulldomain:
      prod: ${self:custom.domain.domain}
      other: ${self:custom.stage}.${self:custom.domain.domain}
    # This value has been customised so I can maintain multiple demonstration sites
    domain: ${self:custom.postname}.${self:custom.domain.zonename}
    domainname: ${self:custom.domain.fulldomain.${self:custom.stage}, self:custom.domain.fulldomain.other}
    # DNS Zone name (this is only required so I can maintain multiple demonstration sites)
    zonename: alphageek.com.au
    cacheControlMaxAgeHTMLByStage:
      # HTML Cache time for production environment
      prod: 3600
      # HTML Cache time for other environments
      other: 0
    cacheControlMaxAgeHTML: ${self:custom.domain.cacheControlMaxAgeHTMLByStage.${self:custom.stage}, self:custom.domain.cacheControlMaxAgeHTMLByStage.other}
    sslCertificateARN: arn:aws:acm:us-east-1:165657443288:certificate/61d202ea-12f2-4282-b602-9c3b83183c7a
  assets:
    targets:
      # Configuration for HTML files (overriding the default cache control age)
      - bucket:
          Ref: WebsiteS3Bucket
        files:
          - source: ./public/
            headers:
              CacheControl: max-age=${self:custom.domain.cacheControlMaxAgeHTML}
            empty: true
            globs:
              - '**/*.html'
      # Configuration for all assets
      - bucket:
          Ref: WebsiteS3Bucket
        files:
          - source: ./public/
            empty: true
            globs:
              - '**/*.js'
              - '**/*.css'
              - '**/*.jpg'
              - '**/*.png'
              - '**/*.gif'
  # AWS Region to S3 website hostname mapping
  s3DNSName:
    us-east-2: s3-website.us-east-2.amazonaws.com
    us-east-1: s3-website-us-east-1.amazonaws.com
    us-west-1: s3-website-us-west-1.amazonaws.com
    us-west-2: s3-website-us-west-2.amazonaws.com
    ap-south-1: s3-website.ap-south-1.amazonaws.com
    ap-northeast-3: s3-website.ap-northeast-3.amazonaws.com
    ap-northeast-2: s3-website.ap-northeast-2.amazonaws.com
    ap-southeast-1: s3-website-ap-southeast-1.amazonaws.com
    ap-southeast-2: s3-website-ap-southeast-2.amazonaws.com
    ap-northeast-1: s3-website-ap-northeast-1.amazonaws.com
    ca-central-1: s3-website.ca-central-1.amazonaws.com
    eu-central-1: s3-website.eu-central-1.amazonaws.com
    eu-west-1: s3-website-eu-west-1.amazonaws.com
    eu-west-2: s3-website.eu-west-2.amazonaws.com
    eu-west-3: s3-website.eu-west-3.amazonaws.com
    eu-north-1: s3-website.eu-north-1.amazonaws.com
    sa-east-1: s3-website-sa-east-1.amazonaws.com
  # Determine what resources file to include based on the current stage
  customConfigFile: ${self:custom.customConfigFiles.${self:custom.stage}, self:custom.customConfigFiles.other}
  customConfigFiles:
    prod: prod
    other: other

# Define the resources we will need to host the site
resources:
  # Include the resources file
  - ${file(config/resources.yml)}
  # Include the outputs file
  - ${file(config/outputs.yml)}
  # Include a custom configuration file based on the environment
  - ${file(config/resources/environment/${self:custom.customConfigFile}.yml)}
package.jsonview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
{
  "name": "hexo-site",
  "version": "0.0.0",
  "private": true,
  "hexo": {
    "version": "3.8.0"
  },
  "devDependencies": {
    "@silvermine/serverless-plugin-cloudfront-lambda-edge": "^2.1.1",
    "broken-link-checker-local": "^0.2.0",
    "hexo": "^3.7.0",
    "hexo-generator-archive": "^0.1.5",
    "hexo-generator-category": "^0.1.3",
    "hexo-generator-index": "^0.2.1",
    "hexo-generator-tag": "^0.2.0",
    "hexo-renderer-ejs": "^0.3.1",
    "hexo-renderer-marked": "^0.3.2",
    "hexo-renderer-stylus": "^0.3.3",
    "hexo-server": "^0.3.1",
    "jest": "^24.0.0",
    "serverless-lambda-version": "^0.1.2",
    "serverless-plugin-scripts": "^1.0.2",
    "serverless-s3-deploy": "^0.8.0"
  },
  "scripts": {
    "jest": "jest",
    "mdlint-drafts": "markdownlint --config .mdconfig ./source/_drafts",
    "mdlint": "markdownlint --config .mdconfig ./source/_posts",
    "precheck": "npm run jest && npm run mdlint",
    "clean": "hexo clean",
    "build": "hexo generate",
    "cleanbuild": "npm run clean && npm run build",
    "linkcheck": "blcl --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ public",
    "htmlcheck": "node-w3c-validator -v -s -i public/",
    "a11ycheck": "pa11y-ci public/*.html public/*/*.html public/*/*/*.html public/*/*/*/*.html public/*/*/*/*/*.html || true",
    "validate": "npm run linkcheck && npm run htmlcheck && npm run a11ycheck",
    "buildtest-local": "npm run precheck && npm run cleanbuild && npm run validate",
    "precommit": "npm run buildtest-local",
    "deploy-uat-infra": "npx serverless deploy -s dev",
    "deploy-uat-site": "npx serverless s3deploy -s dev -v",
    "deploy-uat": "npm run deploy-uat-infra && npm run deploy-uat-site",
    "linkcheck-uat": "npx blc --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ --user-agent '**PASSWORD_DEFINED_IN_SERVERLESS_CONFIGURATION** Tester' http://dev.alphageek.com.au",
    "test-uat": "npm run deploy-uat && npm run linkcheck-uat",
    "buildtest-uat": "npm run buildtest-local && test-uat",
    "deploy-prod-infra": "npx serverless deploy -s prod",
    "deploy-prod-site": "npx serverless s3deploy -s prod -v",
    "deploy-prod": "npm run deploy-prod-infra && npm run deploy-prod-site",
    "linkcheck-prod": "npx blc --filter-level 3 --get --recursive --exclude /atom.xml --exclude /favicon.png --exclude http://2019-01-28-securing-s3.demo.alphageek.com.au.s3-website-us-east-1.amazonaws.com --exclude http://localhost:4000 --exclude http://dev./%3Cyour_domain%3E/ http://alphageek.com.au",
    "test-prod": "npm run deploy-prod && npm run linkcheck-prod",
    "buildtest-prod": "npm run buildtest-uat && test-prod"
  },
  "jest": {
    "testPathIgnorePatterns": [
      "/node_modules/",
      "/source/code/",
      "/public/"
    ]
  }
}
.mdconfigview raw
1
2
3
4
5
6
7
8
9
{
  "MD002": {
    "level": 2
  },
  "MD013": false,
  "MD026": {
    "punctuation": ".,;:!"
  }
}
.semaphore/semaphore.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
version: v1.0
name: Hexo Serverless Build Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Prepare the build environment
  - name: Prepare
    task:
      jobs:
        # Make sure we've got the right Java version configured
        - name: Set Java Version
          commands:
            - change-java-version 8
        # Run NPM install, using Semaphore's cache where possible
        - name: NPM Install
          commands:
            # Update NPM because it's so old
            - cache restore npm
            - npm i -g npm
            - cache store npm .nvm/versions/node/v8.11.3/lib/node_modules/npm
            - checkout
            # Reuse dependencies from cache and avoid installing them from scratch:
            - cache restore node-modules-$(checksum package-lock.json)
            - npm ci
            - cache store node-modules-$(checksum package-lock.json) node_modules
  # Run the validation routines that don't require a build
  - name: Validate
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Run the jest test suite
        - name: Run Jest Tests
          commands:
            - npm run jest
        # Run the Markdown linter
        - name: MD Lint
          commands:
            - npm run mdlint
  # Build the deployment files
  - name: Build
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Ensure we have a clean build directory, generate the files and add asset versioning
        - name: Build Site
          commands:
            - npm run clean
            - npm run build
            - cache store public-$(find source -type f -exec cat {} + | checksum) public
  # Run tests on the deployment files
  - name: Test Locally
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
          - cache restore public-$(find source -type f -exec cat {} + | checksum)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck
        # Check that the HTML is valid
        - name: Test W3C compatability
          commands:
            - npm run htmlcheck
        # Check if we meet a11y standards
        - name: Test Accessibility
          commands:
            - npm run a11ycheck
promotions:
  - name: Deploy Content to UAT
    pipeline_file: uat-content.yml
    auto_promote_on:
      - result: passed
        branch:
          - develop
  - name: Deploy Infra to UAT
    pipeline_file: uat-infra.yml
  - name: Deploy Content to Production
    pipeline_file: prod-content.yml
    auto_promote_on:
      - result: passed
        branch:
          - master
  - name: Deploy Infra to Production
    pipeline_file: prod-infra.yml
.semaphore/prod-content.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek Production Content Deployment Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Use Serverless to deploy to Production
  - name: Publish to Prod
    task:
      # Import the secret environment variables
      secrets:
        - name: alphageek-aws
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
          - cache restore public-$(find source -type f -exec cat {} + | checksum)
      jobs:
        - name: Deploy Content
          commands:
            - npm run deploy-prod-site
  # Run tests on the Production site
  - name: Test Production
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck-prod
.semaphore/prod-infra.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek Production Infrastructure Deployment Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Use serverless to deploy to Production
  - name: Publish to Production
    task:
      # Import the secret environment variables
      secrets:
        - name: alphageek-aws
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        - name: Deploy Infrastructure
          commands:
            - npm run deploy-prod-infra
            - cache store serverless-$SEMAPHORE_GIT_BRANCH .serverless
  # Run tests on the UAT site
  - name: Test Production
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck-prod
.semaphore/uat-content.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek UAT Content Deployment Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Use Serverless to deploy to UAT
  - name: Publish to UAT
    task:
      # Import the secret environment variables
      secrets:
        - name: alphageek-aws
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
          - cache restore public-$(find source -type f -exec cat {} + | checksum)
      jobs:
        - name: Deploy Content
          commands:
            - npm run deploy-uat-site
  # Run tests on the UAT site
  - name: Test UAT
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck-uat
.semaphore/uat-infra.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
version: v1.0
name: AlphaGeek UAT Infrastructure Deployment Pipeline
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  # Use serverless to deploy to UAT
  - name: Publish to UAT
    task:
      # Import the secret environment variables
      secrets:
        - name: alphageek-aws
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        - name: Deploy Infrastructure
          commands:
            - npm run deploy-uat-infra
            - cache store serverless-$SEMAPHORE_GIT_BRANCH .serverless
  # Run tests on the UAT site
  - name: Test UAT
    task:
      prologue:
        commands:
          - checkout
          - cache restore npm
          - cache restore node-modules-$(checksum package-lock.json)
      jobs:
        # Check that all links are valid
        - name: Test link validity
          commands:
            - npm run linkcheck-uat
config/resources.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
Resources:
  # Set-up an S3 bucket to store the site
  WebsiteS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: BucketOwnerFullControl
      BucketName: ${self:custom.domain.domainname}
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
  # Set-up a policy on the bucket so it can be used as a website
  WebsiteBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      PolicyDocument:
        Id:
          Fn::Join:
            - ""
            - - ${self:service.name}
              - BucketPolicy
        Statement:
          - Sid: CloudFrontForGetBucketObjects
            Effect: Allow
            Principal:
              CanonicalUser:
                Fn::GetAtt:
                  - CloudFrontIdentity
                  - S3CanonicalUserId
            Action: 's3:GetObject'
            Resource:
              Fn::Join:
                - ''
                -
                  - 'arn:aws:s3:::'
                  - Ref: WebsiteS3Bucket
                  - /*
      Bucket:
        Ref: WebsiteS3Bucket
  # Configure CloudFront to get all content from S3
  WebsiteCloudFrontDistribution:
    Type: 'AWS::CloudFront::Distribution'
    Properties:
      DistributionConfig:
        WebACLId:
          Ref: CustomAuthorizationHeaderRestriction
        Aliases:
          - ${self:custom.domain.domainname}
          - www.${self:custom.domain.domainname}
        CustomErrorResponses:
          - ErrorCode: '404'
            ResponsePagePath: "/error.html"
            ResponseCode: '200'
            ErrorCachingMinTTL: '30'
        DefaultCacheBehavior:
          Compress: true
          ForwardedValues:
            QueryString: false
            Cookies:
              Forward: all
          SmoothStreaming: false
          TargetOriginId: defaultOrigin
          ViewerProtocolPolicy: redirect-to-https
        DefaultRootObject: index.html
        Enabled: true
        Origins:
          - DomainName:
              Fn::GetAtt:
                - WebsiteS3Bucket
                - DomainName
            Id: defaultOrigin
            S3OriginConfig:
              OriginAccessIdentity:
                Fn::Join:
                  - "/"
                  - - origin-access-identity
                    - cloudfront
                    - Ref: CloudFrontIdentity
        PriceClass: PriceClass_All
        ViewerCertificate:
          AcmCertificateArn: ${self:custom.domain.sslCertificateARN}
          SslSupportMethod: sni-only
  # DNS Record for the domain
  WebsiteDNSRecord:
    Type: "AWS::Route53::RecordSet"
    Properties:
      AliasTarget:
        DNSName:
          Fn::GetAtt:
            - WebsiteCloudFrontDistribution
            - DomainName
        HostedZoneId: Z2FDTNDATAQYW2
      HostedZoneName: ${self:custom.domain.domain}.
      Name: ${self:custom.domain.domainname}
      Type: 'A'
  # DNS Record for www.domain
  WebsiteWWWDNSRecord:
    Type: "AWS::Route53::RecordSet"
    Properties:
      AliasTarget:
        DNSName:
          Fn::GetAtt:
            - WebsiteCloudFrontDistribution
            - DomainName
        HostedZoneId: Z2FDTNDATAQYW2
      HostedZoneName: ${self:custom.domain.domain}.
      Name: www.${self:custom.domain.domainname}
      Type: 'A'
  # Predicate to match the authorization header
  CustomAuthorizationHeader:
    Type: AWS::WAF::ByteMatchSet
    Properties:
      ByteMatchTuples:
        -
          FieldToMatch:
            Type: HEADER
            Data: Authorization
          TargetString:
            Fn::Join:
              - " "
              - - Custom
                - "**Password**"
          TextTransformation: NONE
          PositionalConstraint: EXACTLY
      Name:
        Fn::Join:
          - "_"
          - - ${self:custom.domain.domainname}
            - Authorization
            - Header
  CustomAuthorizationHeaderRule:
    Type: AWS::WAF::Rule
    Properties:
      Name:
        Fn::Join:
          - "_"
          - - ${self:custom.domain.domainname}
            - Authorization
            - Header
            - Rule
      MetricName:
        Fn::Join:
          - ""
          - - ${self:custom.stage}
            - ${self:service.name}
            - Authorization
            - Header
            - Rule
      Predicates:
        -
          DataId:
            Ref: CustomAuthorizationHeader
          Negated: false
          Type: ByteMatch
  CustomUserAgentHeader:
    Type: AWS::WAF::ByteMatchSet
    Properties:
      ByteMatchTuples:
        -
          FieldToMatch:
            Type: HEADER
            Data: User-Agent
          TargetString:
            Fn::Join:
              - " "
              - - ${self:custom.security.passwords.development}
                - "Tester"
          TextTransformation: NONE
          PositionalConstraint: EXACTLY
      Name:
        Fn::Join:
          - "_"
          - - ${self:custom.domain.domainname}
            - UserAgent
            - Header
  CustomUserAgentHeaderRule:
    Type: AWS::WAF::Rule
    Properties:
      Name:
        Fn::Join:
          - "_"
          - - ${self:custom.domain.domainname}
            - UserAgent
            - Header
            - Rule
      MetricName:
        Fn::Join:
          - ""
          - - ${self:custom.domain.domain}
            - ${self:custom.stage}
            - UserrAgent
            - Header
            - Rule
      Predicates:
        -
          DataId:
            Ref: CustomUserAgentHeader
          Negated: false
          Type: ByteMatch
config/other.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Resources:
  # Require the custom authorisation header with the correct password in non-production environment
  CustomAuthorizationHeaderRestriction:
    Type: AWS::WAF::WebACL
    Properties:
      DefaultAction:
        Type: BLOCK
      Name:
        Fn::Join:
          - "_"
          - - ${self:custom.domain.domainname}
            - Authorization
            - Header
            - Restriction
      MetricName:
        Fn::Join:
          - ""
          - - ${self:custom.stage}
            - ${self:service.name}
            - Authorization
            - Header
            - Restriction
      Rules:
        -
          Action: ALLOW
          Priority: 1
          RuleId:
            Ref: CustomAuthorizationHeaderRule
        -
          Action:
            Type: ALLOW
          Priority: 2
          RuleId:
            Ref: CustomUserAgentHeaderRule
test/functions/urlRewrite.test.jsview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
// Load the file to test
const urlRewriteTest = require('../../functions/urlRewrite');
// Load some data that can be reused for other lambda@Edge functions
const lambdaAtEdgeFixture = require('../fixtures/lambdaAtEdge');

test(
    'url-rewrite handler appends index.html to root request',
    () => {
        const expectedResponse = { uri: lambdaAtEdgeFixture.event.root_object.Records[0].cf.request.uri + 'index.html' };
        expect(
            urlRewriteTest.handler(
                lambdaAtEdgeFixture.event.root_object,
                lambdaAtEdgeFixture.context.webClient,
                lambdaAtEdgeFixture.callback
            )
        ).toEqual(expect.objectContaining(expectedResponse));
    }
);

test(
    'url-rewrite handler appends index.html to directory requests',
    () => {
        const expectedResponse = { uri: lambdaAtEdgeFixture.event.subdirectory.Records[0].cf.request.uri + 'index.html' };
        expect(
            urlRewriteTest.handler(
                lambdaAtEdgeFixture.event.subdirectory,
                lambdaAtEdgeFixture.context.webClient,
                lambdaAtEdgeFixture.callback
            )
        ).toEqual(expect.objectContaining(expectedResponse));
    }
);

test(
    'url-rewrite handler does not append index.html to root-directory requests',
    () => {
        const expectedResponse = { uri: lambdaAtEdgeFixture.event.file.Records[0].cf.request.uri };
        expect(
            urlRewriteTest.handler(
                lambdaAtEdgeFixture.event.file,
                lambdaAtEdgeFixture.context.webClient,
                lambdaAtEdgeFixture.callback
            )
        ).toEqual(expect.objectContaining(expectedResponse));
    }
);

test(
    'url-rewrite handler does not append index.html to non-directory requests',
    () => {
        const expectedResponse = { uri: lambdaAtEdgeFixture.event.file_in_subdirectory.Records[0].cf.request.uri };
        expect(
            urlRewriteTest.handler(
                lambdaAtEdgeFixture.event.file_in_subdirectory,
                lambdaAtEdgeFixture.context.webClient,
                lambdaAtEdgeFixture.callback
            )
        ).toEqual(expect.objectContaining(expectedResponse));
    }
);
config/outputs.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Outputs:
  WebsiteURL:
    Value:
      Fn::GetAtt:
        - WebsiteS3Bucket
        - WebsiteURL
    Description: URL for my website hosted on S3
  S3BucketSecureURL:
    Value:
      Fn::Join:
        - ''
        -
          - 'https://'
          - Fn::GetAtt:
              - WebsiteS3Bucket
              - DomainName
    Description: Secure URL of S3 bucket to hold website content
config/prod.ymlview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Resources:
  # Allow the custom authorisation header in the production environment
  CustomAuthorizationHeaderRestriction:
    Type: AWS::WAF::WebACL
    Properties:
      DefaultAction:
        Type: ALLOW
      Name:
        Fn::Join:
          - "_"
          - - ${self:custom.domain.domainname}
            - Authorization
            - Header
            - Restriction
      MetricName:
        Fn::Join:
          - ""
          - - ${self:custom.stage}
            - ${self:service.name}
            - Authorization
            - Header
            - Restriction
      Rules:
        -
          Action:
            Type: ALLOW
          Priority: 1
          RuleId:
            Ref: CustomAuthorizationHeaderRule

# Example Site