API fun with Jeopardy! Access >100k Jeopardy clues scraped from j-archive via a simple api.
The api is backed by a go web server built with gin that exposes a few endpoints to access historical Jeopardy data.
The shape of the data returned from the api aligns with the db schema, this is accomplished via various struct tags on the type definitions.
db
tag is used by the sqlx library to map the db columns to the struct fieldsjson
tag is used by the gin library to map the struct fields to the json responseexample
tag is used by the swaggo library to generate example responses for the swagger docsform
and binding
tags are used by gin to map query arguments to a struct with some basic validationfor example, the pkg.models.Clue
type is defined as follows:
type Clue struct {
ClueID int64 `db:"clue_id" json:"clueId" example:"804002032"`
GameID int64 `db:"game_id" json:"gameId" example:"8040"`
CategoryID int64 `db:"category_id" json:"categoryId" example:"804092001"`
Question string `db:"question" json:"question" example:"This is the question."`
Answer string `db:"answer" json:"answer" example:"This is the answer."`
}
pkg.server.Filter
type:// Filter describes the query parameters that can be used to filter the results of an API query.
type Filter struct {
Random *bool `form:"random"`
ID *int64 `form:"id"`
Page *int64 `form:"page,default=0" binding:"min=0"`
Limit *int64 `form:"limit,default=1" binding:"min=1,max=100"`
}
/
endpoint and is an html template that displays the swagger docs, somesources
causing a watched task to continuously rebuild because of some circular dependencies.Currently the app uses a file based sqlite database. Below are some notes on the deprecated mysql setup. All in all, the 15 seasons of data currently in the DB only end up as ~25 MB .sql file. Using sqlite removed the need to run a mysql server and made the app easier to deploy and test.
Getting the data into the database started as a manual process, and hasn't been automated yet because the data is all there and I haven't needed to import / export it recently.
Here's how I went about doing it initially:
DB_HOST
, DB_USER
, DB_PASS
, DB_NAME
environment variables to target a mariadb/mysql
server running in my home lab.task sql:dump
command will create a dump of the database defined by the environment variables and write it to data/dump.sql.gz
.note: all the scraping was done against the mysql databse, not the current sqlite setup (though I did some brief testing and things seemed to still work for the most part ymmv)
The scraper package contains the code to scrape j-archive for jeopardy clues and write the data to a mysql database. Colly is the package use to scrape the data and sqlx is used to write the data to the db. The scraping happened in a few passes, more or less following these steps:
Get all the seasons and populate the seasons table.
Get all the games for each season and populate the game table.
Get all the clues for each game in each season and populate the category and clue tables
<table>
elements on the page