⏱ Zei - 🏢 Sca - 😺 Git

zeit/now - scaphold - github

A step-by-step walk-through to build a little service that makes use of modern, open technologies.

💡 Concept

Problem

The problem that we try to solve today is that Github doesn't show the public teams to non-organization members.

For example: if you are logged-out of Github (or you are not a member of NodeSchool) and go to the NodeSchool page : it will not show the "Teams" tab.

For making the community clearer and giving recognition to the members in the team we want to make the public teams visible to everyone.

Solution

Let's write a little Node.js server that fetches the teams from Github and stores them in a database. This server then offers the data to the public.

CPU & Memory

Zeit/now offers free CPU for open-source projects. Not just that: it will come in handy that setting-up process is very simple.

Storage

Instead of a common REST API, we try to go with the times and provide a fancier GraphQL API. This is more flexible and perhaps faster.

Luckily Scaphold offers a free GraphQL database as a service.

😺 Github API

If you used the Github API before, you might be all like "been-there-done-that", but Github has an early-access GraphQL API which makes this a little more exciting.

To access the API through Node, first init the project zei-sca-git: (Note: the installation of Node.js 7 & Git is assumed)

$ mkdir zei-sca-git; cd zei-sca-git
$ npm init -y
$ git init

Then we take care of common node setup steps:

1. Add "private": true to the package.json to make sure we don't accidentally publish it prematurely.

2. Add author, description and keyword metadata to the package.json to be clear about our intent.

3. Make sure that the node_modules folder is part of .gitignore

   $ echo "node_modules" >> .gitignore
   

Then finish our commit:

$ git add .; git commit -m "initial commit"

Use the GraphQL Github API in Node

Lets first install the graphql-fetch package and add tap to test our progress.

$ npm install --save graphql-fetch isomorphic-fetch
$ npm install --save-dev tap

(graphql-fetch requires isomorphic-fetch)

The GraphQL specification for Github is available at https://api.github.com/graphql , so we can prepare the fetch call like this:

(lib/loadTeams.js)

const fetch = require('graphql-fetch')('https://api.github.com/graphql')

We will also need a Github Token . As Placeholder let me use 'ABCGithubIsSweetXYZ' in the examples.

const GITHUB_TOKEN = 'ABCGithubIsSweetXYZ'

We can then use the documentation available here to immediately test the API online.

With some reading of docs we learn to fetch an organization ID for a login like this:

module.exports = (login) =>
  fetch(`
    query Organization ($login: String!) {
      organization(login: $login) {
        id
      }
    }
  `, { login }, {
    headers: new Headers({
      Authorization: `bearer ${GITHUB_TOKEN}`
    })
  })

With tap we then simple can write tests to check if the ID is correct:

(../test/loadTeams.js)

const { test } = require('tap')
const loadTeams = require('../lib/loadTeams.js')

test('get organization id', t =>
  loadTeams('nodeschool').then(result => {
    t.equals(result.data.organization.id, 'MDEyOk9yZ2FuaXphdGlvbjU0Mzc1ODc=')
  })
)

Now that we have the code and the tests, we just need to add the test script to the package.json:

"scripts": {
  "test": "tap -- test/**"
}

With this, we can run $ npm test and it should be green.

🛠 Prepare the Database

Since we already use GraphQL to access Github, we can go all the way and use GraphQL to store our data as well! Recently, several "GraphQL-as-a-Service" databases were launched. Scaphold is our choice for this experiment, so let's create an account!

With the new account we can further create an app:

Now that we have created the app, we can specify the Types we would like to store. If you have specified MySQL tables before then this might feel familiar:

For today's work we need a schema that looks like ./scaphold.schema (./scaphold.schema.json is an export of the schema I created in preparation).

Now that we also have the data schema, we can use the app to store our teams! Scaphold offers a direct link to access our data storage:

Scaphold also allows us to immediately explore the API to the data storage with iGraphQL.

Now we have a database! 🎉

🔒 Security

By default all data in Scaphold is unprotected. But it is possible to add a setting that allows the modification of data only to admin users.

Limit the permissions for Everyone to read.

With this setting active, only admin users can edit the data.

💾 Store some data

We can use GraphQL not just to request data. We can also change it! In GraphQL that is called a "Mutation".

For our test we want to be able to modify everything, specially the protected parts, so we have to get an Admin Token first.

As placeholder let me use 'ABCScapholdForTheWinXYZ' in the following code.

Now, Let's start using the GraphQL mutation API:

(lib/teamCRUD.js)

require('isomorphic-fetch')

const fetch = require('graphql-fetch')(
  'https://us-west-2.api.scaphold.io/graphql/zei-sca-git'
)
const SCAPHOLD_TOKEN = 'ABCScapholdForTheWinXYZ'
const HEADERS = {
  headers: new Headers({
    Authorization: `bearer ${SCAPHOLD_TOKEN}`
  })
}

exports.create = team =>
  fetch(`
    mutation CreateTeam($team: CreateTeamInput!) {
      createTeam(input: $team) {
        changedTeam {
          id
          slug
          privacy
          name
        }
      }
    }
  `, { team }, HEADERS).then(result => result.data && result.data.createTeam.changedTeam)

exports.read = id =>
  fetch(`
    query ReadTeam($id: ID!) {
      getTeam(id: $id) {
        id
        slug
        privacy
        name
      }
    }
  `, { id }, HEADERS).then(result => result.data && result.data.getTeam)

exports.update = team =>
  fetch(`
    mutation UpdateTeam($team: UpdateTeamInput!) {
      updateTeam(input: $team) {
        changedTeam {
          id
          slug
          privacy
          name
        }
      }
    }
  `, { team }, HEADERS).then(result => result.data && result.data.updateTeam.changedTeam)

exports.del = id =>
  fetch(`
    mutation DeleteTeam($team: DeleteTeamInput!) {
      deleteTeam(input: $team) {
        changedTeam { slug }
      }
    }
  `, { team: { id } }, HEADERS).then(result => (result.data && result.data.deleteTeam) ? true : false)

With those 4 methods we have a complete CRUD API. Let's immediately test this!

(test/teamCRUD.js)

const { test } = require('tap')
const { create, read, update, del } = require('../lib/teamCRUD.js')

test('create, read, update and delete a team', t => {
  let id
  let slug = 'abcd' + Math.random().toString(32)
  return create({
    name: 'ABCD',
    slug: slug,
    privacy: 'VISIBLE'
  }).then(team => {
    t.notEquals(team, null)
    t.equals(team.name, 'ABCD')
    t.equals(team.slug, slug)
    t.equals(team.privacy, 'VISIBLE')
    t.notEquals(team.id)
    id = team.id
    return read(id)
  }).then(team => {
    t.notEquals(team, null)
    t.equals(team.id, id)
    t.equals(team.name, 'ABCD')
    team.name = 'EFGH'
    return update(team)
  }).then(team => {
    t.notEquals(team, null)
    t.equals(team.id, id)
    t.equals(team.name, 'EFGH')
    return read(id)
  }).then(team => {
    t.notEquals(team, null)
    t.equals(team.id, id)
    t.equals(team.name, 'EFGH')
    return del(id)
  }).then(successful => {
    t.equals(successful, true)
    return read(id)
  }).then(team => {
    t.equals(team, null)
  })
})

Yeah! Now we can store data! 🎉

🚀 Setup a Server

Now that we have storage and purpose we need to start using it! Before we do that though, let's first move all our keys to environment variables 😅. If we keep them in the code they might become visible to everyone, and that would be bad.

🔑 Preparation: Environment variables

We can easily use process.env to get the GITHUB_TOKEN and SCAPHOLD_TOKEN like this:

const GITHUB_TOKEN = process.env.GITHUB_TOKEN
if (!GITHUB_TOKEN) {
  throw new Error('Please specify the GITHUB_TOKEN environment variable')
}

after this change we need to run the tests like this:

$ env GITHUB_TOKEN=ABCGithubIsSweetXYZ \
      SCAPHOLD_TOKEN=ABCScapholdForTheWinXYZ \
      npm test

🏴 Setup a simple Server

We still need a little server for the sync process. For this project http.createServer that comes with Node.js is powerful enough.

(index.js)

const { createServer } = require('http')
const SECRET = '/secret'
const server = createServer((req, res) => {
  if (req.url === SECRET && req.method === 'GET') {
    res.writeHead(200, {'Content-Type': 'text/plain'})
    res.end('you found me!')
    return
  }
  res.writeHead(404, {'Content-Type': 'text/plain'})
  res.end('not found')
})
server.listen( () => {
  const address = server.address()
  const host = address.address === '::' ? 'localhost' : address.address
  console.log(`Server started on http://${host}:${address.port}`)
})

With node index.js we can start the server and it will show a 404 error message for all pages except /secret (/secret will show you found me!).

To clean it up a little we should add a start script to the package.json:

"scripts": {
  "test": "tap -- test/**",
  "start": "node index.js"
}

With this script we can start the same server using npm start.

🏳 Launch it with now

Zeit/now is PaaS that has computers running in the cloud with Node on them.

now be used easily through a command line tool. We need to install this tool using:

$ npm install now -g

(Note: some setups need to run this with sudo npm install now -g)

Then you can start the server simply using:

$ now

Since I deployed this server you can look it up immediately here:

https://zei-sca-git-iirtkazzso.now.sh/

It will output not found. Accessing: /secret will show you found me!.

https://zei-sca-git-iirtkazzso.now.sh/secret

You can immediately look at the source code here:

https://zei-sca-git-iirtkazzso.now.sh/_src/?f=index.js

🏁 Connect the parts

All parts are working separately but we need to get them to work together.

1.) Get all data

./lib/loadTeams.js does not load all the data of teams yet. So, lets change the API to return the full team object:

(lib/loadTeams.js)

query Organization ($login: String!) {
  organization(login: $login) {
    id
    teams(first: 30) {
      pageInfo {
        hasNextPage
        endCursor
      }
      edges {
        node {
          name
          privacy
          slug
        }
      }
    }
  }
}

Now we get the team data with every response.

2.) Write the data to the server

When the server gets to the secret path it should start the sync process:

(index.js)

const sync = require('./lib/sync.js')
const GITHUB_ORG = 'nodeschool'
// ...
  if (req.url === SECRET && req.method === 'GET') {
    sync(GITHUB_ORG).then(() => {
      //..
    }).catch((e) => {
      res.write(500, {'Content-Type': 'text/plain'})
      res.end(e.stack || e.toString())
    })
    return
  }

For a the start, lets just store the teams. To make our lives a little easier still, lets also use bluebird.

$ npm i bluebird --save

(lib/sync.js)

const { map } = require('bluebird')
const loadTeams = require('./loadTeams.js')
const { create } = require('./teamCRUD.js')
const running = {}

module.exports = login => {
  if (running[login]) {
    // Return currently running process if not-yet finished, DDOS prevention
    return running[login]
  }
  return running[login] = loadTeams(login)
    .then((result) => {
      var teams = result
        .data.organization.teams.edges
        // We only need the node
        .map(edge => edge.node)
        // Hide secret teams
        .filter(team => team.privacy !== 'SECRET')

      return map(
        teams,
        // Eat error messages
        team => create(team).catch(e => null),
        // Limit the requests to 5 parallel, to prevent us accidentally DDOSing
        // Scaphold
        { concurrency: 5 }
      )
    })
    .then(result => {
      delete running[login]
      return result
    })
}

Accessing the server at /secret will sync all the teams from Github to Scaphold.

3.) Start with secrets

We have to extract the GITHUB_ORG configuration constant and the SECRET constant same like we did with GITHUB_TOKEN to make sure that only people we know can update the database.

const SECRET = '/' + process.env.SECRET
if (!SECRET) {
  throw new Error('Please specify the SECRET environment variable')
}

After that we need 4 variables to start our server:

• SECRET
• GITHUB_ORG
• GITHUB_TOKEN
• SCAPHOLD_TOKEN

Abd we can easily start now with those variables.

$ now -e SECRET=secret \
      -e GITHUB_ORG=nodeshool \
      -e GITHUB_TOKEN=ABCGithubIsSweetXYZ \
      -e SCAPHOLD_TOKEN=ABCScapholdForTheWinXYZ

The server should be running and we can sync the data simply by opening:

$ curl https://zei-sca-git-yhyfypszsj.now.sh/secret

🤓 We did it! High Five! ✋🏽

You can explore the public API here.

🤔 Summary

GraphQL as a method is bound to make our lives easier. The transportable schema definitions and new GraphQL services like Scaphold work surprisingly well. Combined with a　microservice architecture like Zeit/now we should be able to create quickly, very useful open database while being much less dependent on the whims of Proprietary technologies. 🤑

All this code is hosted on Github.

I hope you enjoyed it!

One final note: For NodeSchool, it would be super awesome if we could turn this into concept or prototype into a full-fletched syncing server.