A little tutorial to learn zeit/now, scaphold and github api.
A step-by-step walk-through to build a little service that makes use of modern, open technologies.
The problem that we try to solve today is that Github doesn't show the public teams to non-organization members.
For example: if you are logged-out of Github (or you are not a member of NodeSchool) and go to the NodeSchool page : it will not show the "Teams" tab.
For making the community clearer and giving recognition to the members in the team we want to make the public teams visible to everyone.
Let's write a little Node.js server that fetches the teams from Github and stores them in a database. This server then offers the data to the public.
Zeit/now offers free CPU for open-source projects. Not just that: it will come in handy that setting-up process is very simple.
Instead of a common REST API, we try to go with the times and provide a fancier GraphQL API. This is more flexible and perhaps faster.
Luckily Scaphold offers a free GraphQL database as a service.
If you used the Github API before, you might be all like "been-there-done-that", but Github has an early-access GraphQL API which makes this a little more exciting.
To access the API through Node, first init the project zei-sca-git
:
(Note: the installation of Node.js 7 & Git is assumed)
$ mkdir zei-sca-git; cd zei-sca-git
$ npm init -y
$ git init
Then we take care of common node setup steps:
Add "private": true
to the package.json
to make sure we don't accidentally
publish it prematurely.
Add author
, description
and keyword
metadata to the package.json
to
be clear about our intent.
Make sure that the node_modules
folder is part of .gitignore
$ echo "node_modules" >> .gitignore
Then finish our commit:
$ git add .; git commit -m "initial commit"
Lets first install the graphql-fetch
package and add tap
to test our progress.
$ npm install --save graphql-fetch isomorphic-fetch
$ npm install --save-dev tap
(graphql-fetch
requires isomorphic-fetch
)
The GraphQL specification for Github is available at https://api.github.com/graphql , so we can prepare the fetch call like this:
(lib/loadTeams.js)
const fetch = require('graphql-fetch')('https://api.github.com/graphql')
We will also need a Github Token
. As Placeholder let me use 'ABCGithubIsSweetXYZ'
in the examples.
const GITHUB_TOKEN = 'ABCGithubIsSweetXYZ'
We can then use the documentation available here to immediately test the API online.
With some reading of docs we learn to fetch an organization ID for a login
like this:
module.exports = (login) =>
fetch(`
query Organization ($login: String!) {
organization(login: $login) {
id
}
}
`, { login }, {
headers: new Headers({
Authorization: `bearer ${GITHUB_TOKEN}`
})
})
With tap
we then simple can write tests to check if the ID is correct:
(../test/loadTeams.js)
const { test } = require('tap')
const loadTeams = require('../lib/loadTeams.js')
test('get organization id', t =>
loadTeams('nodeschool').then(result => {
t.equals(result.data.organization.id, 'MDEyOk9yZ2FuaXphdGlvbjU0Mzc1ODc=')
})
)
Now that we have the code and the tests, we just need to add the test script
to the package.json
:
"scripts": {
"test": "tap -- test/**"
}
With this, we can run $ npm test
and it should be green.
Since we already use GraphQL to access Github, we can go all the way and use GraphQL to store our data as well! Recently, several "GraphQL-as-a-Service" databases were launched. Scaphold is our choice for this experiment, so let's create an account!
With the new account we can further create an app:
Now that we have created the app, we can specify the Types
we would like to store.
If you have specified MySQL tables before then this might feel familiar:
For today's work we need a schema that looks like ./scaphold.schema
(./scaphold.schema.json
is an export of the schema
I created in preparation).
Now that we also have the data schema, we can use the app to store our teams! Scaphold offers a direct link to access our data storage:
Scaphold also allows us to immediately explore the API to the data storage with iGraphQL.
Now we have a database! π
By default all data in Scaphold is unprotected. But it is possible to add a setting that allows the modification of data only to admin users.
Limit the permissions for Everyone
to read
.
With this setting active, only admin users can edit the data.
We can use GraphQL not just to request data. We can also change it! In GraphQL that is called a "Mutation".
For our test we want to be able to modify everything, specially the protected parts,
so we have to get an Admin Token
first.
As placeholder let me use 'ABCScapholdForTheWinXYZ'
in the following code.
Now, Let's start using the GraphQL mutation API:
(lib/teamCRUD.js)
require('isomorphic-fetch')
const fetch = require('graphql-fetch')(
'https://us-west-2.api.scaphold.io/graphql/zei-sca-git'
)
const SCAPHOLD_TOKEN = 'ABCScapholdForTheWinXYZ'
const HEADERS = {
headers: new Headers({
Authorization: `bearer ${SCAPHOLD_TOKEN}`
})
}
exports.create = team =>
fetch(`
mutation CreateTeam($team: CreateTeamInput!) {
createTeam(input: $team) {
changedTeam {
id
slug
privacy
name
}
}
}
`, { team }, HEADERS).then(result => result.data && result.data.createTeam.changedTeam)
exports.read = id =>
fetch(`
query ReadTeam($id: ID!) {
getTeam(id: $id) {
id
slug
privacy
name
}
}
`, { id }, HEADERS).then(result => result.data && result.data.getTeam)
exports.update = team =>
fetch(`
mutation UpdateTeam($team: UpdateTeamInput!) {
updateTeam(input: $team) {
changedTeam {
id
slug
privacy
name
}
}
}
`, { team }, HEADERS).then(result => result.data && result.data.updateTeam.changedTeam)
exports.del = id =>
fetch(`
mutation DeleteTeam($team: DeleteTeamInput!) {
deleteTeam(input: $team) {
changedTeam { slug }
}
}
`, { team: { id } }, HEADERS).then(result => (result.data && result.data.deleteTeam) ? true : false)
With those 4 methods we have a complete CRUD API. Let's immediately test this!
(test/teamCRUD.js)
const { test } = require('tap')
const { create, read, update, del } = require('../lib/teamCRUD.js')
test('create, read, update and delete a team', t => {
let id
let slug = 'abcd' + Math.random().toString(32)
return create({
name: 'ABCD',
slug: slug,
privacy: 'VISIBLE'
}).then(team => {
t.notEquals(team, null)
t.equals(team.name, 'ABCD')
t.equals(team.slug, slug)
t.equals(team.privacy, 'VISIBLE')
t.notEquals(team.id)
id = team.id
return read(id)
}).then(team => {
t.notEquals(team, null)
t.equals(team.id, id)
t.equals(team.name, 'ABCD')
team.name = 'EFGH'
return update(team)
}).then(team => {
t.notEquals(team, null)
t.equals(team.id, id)
t.equals(team.name, 'EFGH')
return read(id)
}).then(team => {
t.notEquals(team, null)
t.equals(team.id, id)
t.equals(team.name, 'EFGH')
return del(id)
}).then(successful => {
t.equals(successful, true)
return read(id)
}).then(team => {
t.equals(team, null)
})
})
Yeah! Now we can store data! π
Now that we have storage and purpose we need to start using it! Before we do that though, let's first move all our keys to environment variables π . If we keep them in the code they might become visible to everyone, and that would be bad.
We can easily use process.env
to get the GITHUB_TOKEN
and SCAPHOLD_TOKEN
like this:
const GITHUB_TOKEN = process.env.GITHUB_TOKEN
if (!GITHUB_TOKEN) {
throw new Error('Please specify the GITHUB_TOKEN environment variable')
}
after this change we need to run the tests like this:
$ env GITHUB_TOKEN=ABCGithubIsSweetXYZ \
SCAPHOLD_TOKEN=ABCScapholdForTheWinXYZ \
npm test
We still need a little server for the sync process. For this project
http.createServer
that comes with Node.js is powerful enough.
(index.js)
const { createServer } = require('http')
const SECRET = '/secret'
const server = createServer((req, res) => {
if (req.url === SECRET && req.method === 'GET') {
res.writeHead(200, {'Content-Type': 'text/plain'})
res.end('you found me!')
return
}
res.writeHead(404, {'Content-Type': 'text/plain'})
res.end('not found')
})
server.listen( () => {
const address = server.address()
const host = address.address === '::' ? 'localhost' : address.address
console.log(`Server started on http://${host}:${address.port}`)
})
With node index.js
we can start the server and it will show a 404
error message
for all pages except /secret
(/secret
will show you found me!
).
To clean it up a little we should add a start
script to the package.json
:
"scripts": {
"test": "tap -- test/**",
"start": "node index.js"
}
With this script we can start the same server using npm start
.
now
Zeit/now
is PaaS that
has computers running in the cloud with Node on them.
now
be used easily through a command line tool. We need to install this tool
using:
$ npm install now -g
(Note: some setups need to run this with sudo npm install now -g
)
Then you can start the server simply using:
$ now
Since I deployed this server you can look it up immediately here:
https://zei-sca-git-iirtkazzso.now.sh/
It will output not found
. Accessing: /secret
will show you found me!
.
https://zei-sca-git-iirtkazzso.now.sh/secret
You can immediately look at the source code here:
https://zei-sca-git-iirtkazzso.now.sh/_src/?f=index.js
All parts are working separately but we need to get them to work together.
./lib/loadTeams.js
does not load all the data of teams yet. So, lets
change the API to return the full team object:
(lib/loadTeams.js)
query Organization ($login: String!) {
organization(login: $login) {
id
teams(first: 30) {
pageInfo {
hasNextPage
endCursor
}
edges {
node {
name
privacy
slug
}
}
}
}
}
Now we get the team data with every response.
When the server gets to the secret path it should start the sync process:
(index.js)
const sync = require('./lib/sync.js')
const GITHUB_ORG = 'nodeschool'
// ...
if (req.url === SECRET && req.method === 'GET') {
sync(GITHUB_ORG).then(() => {
//..
}).catch((e) => {
res.write(500, {'Content-Type': 'text/plain'})
res.end(e.stack || e.toString())
})
return
}
For a the start, lets just store the teams. To make our lives a little easier still, lets also use bluebird.
$ npm i bluebird --save
(lib/sync.js)
const { map } = require('bluebird')
const loadTeams = require('./loadTeams.js')
const { create } = require('./teamCRUD.js')
const running = {}
module.exports = login => {
if (running[login]) {
// Return currently running process if not-yet finished, DDOS prevention
return running[login]
}
return running[login] = loadTeams(login)
.then((result) => {
var teams = result
.data.organization.teams.edges
// We only need the node
.map(edge => edge.node)
// Hide secret teams
.filter(team => team.privacy !== 'SECRET')
return map(
teams,
// Eat error messages
team => create(team).catch(e => null),
// Limit the requests to 5 parallel, to prevent us accidentally DDOSing
// Scaphold
{ concurrency: 5 }
)
})
.then(result => {
delete running[login]
return result
})
}
Accessing the server at /secret
will sync all the teams from Github to Scaphold.
We have to extract the GITHUB_ORG
configuration constant and the SECRET
constant
same like we did with GITHUB_TOKEN
to make sure that only people we know
can update the database.
const SECRET = '/' + process.env.SECRET
if (!SECRET) {
throw new Error('Please specify the SECRET environment variable')
}
After that we need 4 variables to start our server:
SECRET
GITHUB_ORG
GITHUB_TOKEN
SCAPHOLD_TOKEN
Abd we can easily start now
with those variables.
$ now -e SECRET=secret \
-e GITHUB_ORG=nodeshool \
-e GITHUB_TOKEN=ABCGithubIsSweetXYZ \
-e SCAPHOLD_TOKEN=ABCScapholdForTheWinXYZ
The server should be running and we can sync the data simply by opening:
$ curl https://zei-sca-git-yhyfypszsj.now.sh/secret
π€ We did it! High Five! βπ½
You can explore the public API here.
GraphQL as a method is bound to make our lives easier. The transportable
schema definitions and new GraphQL services like Scaphold work surprisingly well.
Combined with aγmicroservice architecture like Zeit/now
we should be able to
create quickly, very useful open database while being much less dependent on the
whims of Proprietary technologies. π€
All this code is hosted on Github.
I hope you enjoyed it!
One final note: For NodeSchool, it would be super awesome if we could turn this into concept or prototype into a full-fletched syncing server.