Hyper-unspecific data parsing and graphing scripts

A simple Python tool/framework for generating statistics about code in a git repository.

...What?!

I like making random graphs about random stats (code in this case), but got tired of constantly writing new hyper-specific scripts for parsing and graphing that data.

This project aims to solve that by not making any assumptions about the file structure or similar things, and not requiring code changes to collect more stats.

Thanks to @nico for the name :^)

Requirements

• Python 3.11 (see here)
• braceexpand for nicer file glob patterns (pip install -r requirements.txt)
• The following utilities need to be in PATH: git, scc (if used as an
  analyzer)

Usage

Create a config file (TOML):

repository = "/path/to/project"
output = "output/stats.json"

[analyzers]

[analyzers.lines]
type = "scc"

[analyzers.fixmes_and_todos]
type = "grep"
regex = "fixme|todo"
case_insensitive = true

Then call the main.py script with it:

python3 main.py path/to/config.toml

output/stats.json will be created, containing a JSON blob that looks something like this:

[
  {
    "commit": "c048cf2004",
    "timestamp": 1679655523.0,
    "analyzers": {
      "lines": {
        "AK/AllOf.h": {
          "language": "C Header",
          "bytes": 848,
          "lines": 37,
          "code": 25,
          "comment": 5,
          "blank": 7,
          "complexity": 1
        }
        // ...
      },
      "fixmes_and_todos": {
        "Some/File.cpp": 42
        // ...
      }
    }
  }
  // ...
]

That's the hard part done! You can now do whatever you want with this data, here are some ideas...

Scripts

export_csv.py

Not everything likes JSON, but maybe CSV works! Takes all fields of each analyzer and puts them into individual files named <analyzer>.csv in the same directory as the input file.

$ python3 scripts/export_csv.py output/stats.json
$ ls output
fixmes_and_todos.csv lines.csv stats.json
$ cat output/fixmes_and_todos.csv | head -1
c048cf2004,2023-03-24 10:58:43,Some/File.cpp,42
$ cat output/lines.csv | head -1
c048cf2004,2023-03-24 10:58:43,AK/AllOf.h,C Header,848,37,25,5,7,1

This was written specifically with Postgres COPY in mind, and may or may not work for your use case.

plot_grep_analyzer.py

Barebones matplotlib script to plot the sum of grep analyzer results per commit over time.

python3 scripts/plot_grep_analyzer.py output/stats.json fixmes_and_todos

plot_scc_analyzer.py

Matplotlib script to plot a summary of scc results. All lines of each category (blank, comment etc.) are summed together across files.

python3 scripts/plot_scc_analyzer.py output/stats.json lines

Configuring the matplotlib plots

Since matplotlib is used by the plot_* scripts, the design of their plots can be controlled with a matplotlibrc file in the directory from which the script is invoked. For example, these parameters produce academically-looking plots (requires LaTeX):

text.usetex: True
font.size: 17
font.family: serif
axes.grid: True
axes.grid.axis: "y"
axes.grid.which: "both"

Config options

Analyzers

grep

Count occurences of strings or regular expressions across all files. Does not actually use grep(1).

grep_commits

Count occurrences of strings or regular expressions in commit messages.

scc

Uses scc(1) to generate per-file line count stats.

TODO

Tests, more analyzers.

License

MIT.