[opensourcedex]

grimoirelab is a free, open source monitoring & observability project written in Shell and released under GPL-3.0. It has 639 GitHub stars, 234 forks and 78 open issues, and was last pushed 6 days ago. On this registry it ranks #252 of 270 tracked projects in Monitoring & Observability, with 5 head-to-head comparisons available. It gained 12 stars over the last 30 tracked days.

What is grimoirelab?

GrimoireLab is a GPL-3.0 licensed CHAOSS toolset for software development analytics that retrieves data from the systems used to support software development, stores it in databases, enriches it by computing metrics, and makes it available for analytics and visualization, aimed at the people who need to measure development activity rather than guess at it.

What it is

GrimoireLab is a coordinated set of tools, not a single application, and it lives inside the CHAOSS ecosystem. The CHAOSS project develops many of the metrics that GrimoireLab computes, so the numbers the platform produces track a community definition rather than a vendor's private formula. The toolset covers the full chain: retrieval from repositories, storage in databases, enrichment through metric computation, and the analytics and visualization layers on top of the stored data.

The concrete problem it solves is fragmentation. Development data originates in many separate systems, and each one has its own API, its own idea of who a contributor is, and its own export format. GrimoireLab replaces that piecemeal arrangement, in which every source is fetched, stored and enriched by disconnected one-off tooling with no shared metric definitions, with a single pipeline whose stages all speak the same vocabulary and end in one queryable store. Identity consolidation is part of that job: SortingHat, reachable at http://localhost:8000/identities/, manages the identities that the same person accumulates across sources.

Key capabilities

  • Retrieves data from repositories, stores it in databases, enriches it by computing relevant metrics, and exposes it for analytics and visualization as one coordinated toolset.
  • Ships a docker-compose deployment that starts OpenSearch Dashboards at http://localhost:8000 and the OpenSearch API at http://localhost:9200.
  • Provides a default setup in default-grimoirelab-settings that analyzes git activity for the repository itself, so a newcomer can run a real workload immediately.
  • Supports a manual path through the grimoirelab/grimoirelab Docker image, with projects.json and setup-docker.cfg mounted into /home/grimoire/conf/.
  • Manages contributor identities through SortingHat, which is served alongside the dashboards at http://localhost:8000/identities/.
  • Offers importable default dashboards, stored as JSON under panels/json/opensearch_dashboards in grimoirelab-sigils and loaded through Stack Management > Saved Objects.
  • Implements metrics that are developed in part by the CHAOSS project, tying the visualizations to published community metric definitions.

Who uses it and how

  • Project and foundation teams that need development analytics across repositories, with CHAOSS metric definitions as the common ground for reporting.
  • FreeBSD adopted GrimoireLab for Bugzilla analysis and deployed it, which shows the toolset in use outside the ecosystems it was first built around.
  • Small self-hosted deployments: the documented baseline is 2 CPU cores, 8GB of memory, and enough virtual memory mapped for Elasticsearch.
  • Teams that need identity resolution across sources, running SortingHat so that one contributor recorded under several accounts resolves to a single identity.
  • Deployments that treat the dashboards as the starting point and curate their own views by importing and editing the default sigils panels.

Getting started

The README's primary route is to clone https://github.com/chaoss/grimoirelab, change into the docker-compose folder, and run docker-compose up -d, which requires git, a Docker client, Docker Compose, and the memory settings described above. An alternative single-container route runs the grimoirelab/grimoirelab image with docker run, mounting projects.json and setup-docker.cfg and assuming Elasticsearch, Kibana and SortingHat are already running.

How it compares

No paid products that this project replaces are listed in the facts, and the facts name no comparable analytics tools. On the evidence available, GrimoireLab stands alone in this registry.

When to use it — and when not to

A self-hoster must operate more than one moving part: Docker and Docker Compose, OpenSearch or Elasticsearch and Kibana depending on the route chosen, the vm.max_map_count tuning that Elasticsearch requires, and SortingHat if identities are to be resolved. Expect a first-run delay of roughly 10 to 15 minutes even for a small repository, since the pipeline has to fetch and enrich before anything appears in the dashboards. Anyone wanting a hosted, zero-maintenance product, or a single binary with no database behind it, should look elsewhere; the manual docker run path also assumes Elasticsearch, Kibana and SortingHat already exist, so it is not a shortcut past the operational work.

project readme (upstream, from github) — read inline

GrimoireLab

grimoirelab-showcase

GrimoireLab is a CHAOSS toolset for software development analytics. It includes a coordinated set of tools to retrieve data from systems used to support software development (repositories), store it in databases, enrich it by computing relevant metrics, and make it easy to run analytics and visualizations on it.

You can learn more about GrimoireLab in the GrimoireLab tutorial, or visit the GrimoireLab website.

Metrics available in GrimoireLab are, in part, developed in the CHAOSS project. For more information regarding CHAOSS metrics, see the latest release at: https://chaoss.community/metrics/

Getting started

To ease the newcomer experience we are providing a default setup to analyze git activity for this repository. For this set up, there are several options to run GrimoireLab:

Using docker-compose

Requirements:

root@test-68b8628f:~# git --version
git version 2.17.1
root@test-68b8628f:~# docker --version
Docker version 19.03.1, build 74b1e89
root@test-68b8628f:~# docker-compose --version
docker-compose version 1.22.0, build f46880fe

Steps:

  1. Clone this project:
git clone https://github.com/chaoss/grimoirelab
  1. Go to docker-compose folder and run the following command:
cd grimoirelab/docker-compose
docker-compose up -d

Your data will be ready after a while. The waiting time depends on the amount of data to fetch from a repo, for small repositories you can expect your data to be available after 10-15 minutes.

To view the data you can access OpenSearch Dashboards at http://localhost:8000 or use the API directly at http://localhost:9200.

To manage the identities you can access SortingHat at http://localhost:8000/identities/.

By default, OpenSearch Dashboards doesn't contain any visualization, but we have a set of default dashboards. To import them, open OpenSearch Dashboards, go to Stack Management > Saved Objects and import the files you want.

More details or troubleshooting in the docker-compose folder.

Using docker run

Requirements:

root@test-68b8628f:~# git --version
git version 2.17.1
root@test-68b8628f:~# docker --version
Docker version 19.03.1, build 74b1e89
  • Hardware: 2 CPU cores, 8GB memory RAM and set
  • ElasticSearch and Kibana up and running.
  • SortingHat up and running

Steps:

  1. Clone this project:
git clone https://github.com/chaoss/grimoirelab
  1. Go to the project folder and run the following command:
cd grimoirelab
docker run --net=host \
    -v $(pwd)/default-grimoirelab-settings/projects.json:/home/grimoire/conf/projects.json \
    -v $(pwd)/default-grimoirelab-settings/setup-docker.cfg:/home/grimoire/conf/setup.cfg \
    -t grimoirelab/grimoirelab

Your dashboard will be ready after a while at http://localhost:8000. The waiting time depends on the amount of data to fetch from a repo, for small repositories you can expect your data to be visible in the dashboard after 10-15 minutes.

More details in the docker folder.

Using on FreeBSD

Recently FreeBSD has adopted Grimoirelab for their Bugzilla Analysis and have deployed Grimoirelab in their cluster. FreeBSD Foundation has been kind enough to share the process of running Grimoire on FreeBSD which is available here. This guide will also help who are interested in running Grimoirelab from source instead of deploying them with Docker.

Breaking changes

GrimoireLab 1.3.0. SortingHat permission groups.

Starting from GrimoireLab 1.3.0, creating new users in SortingHat requires assigning them to a permission group. By default, they will have read-only permissions. Please refer to the following documentation for instructions on how to update permissions: assign users to permission groups

GrimoireLab components

Currently, GrimoireLab toolkit is organized in the following repositories:

There are also some components built by the GrimoreLab community, which can be useful for you. Other related repositories are:

Contents of this repository

This repository is for content relevant to GrimoireLab as a whole. For example:

  • Issues for new features or bug reports that affect more than one GrimoireLab module. In this case, let's open an issue here, and when implementing the fix or the feature, let´s comment about the specific tickets in the specific modules that are used. For example, when supporting a new datasource, we will need patches (at least) in Perceval, GrimoireELK and panels. In this case, we would open a feature request (or the user story) for the whole case, an issue (and later a pull request) in Perceval for the data retriever, same for GrimoireELK for the enriching code, and same for panels.

  • Release notes for most GrimoireLab components (directory releases).

  • Docker container for showcasing GrimoireLab (directory docker). Includes a Dockerfile and configuration files for the GrimoireLab containers that can be used to demo the technology, and can be the basis for real deployments. See more information in the docker README.md file.

  • If you feel more comfortable with docker-compose, the docker-compose folder includes instructions and configuration files to deploy GrimoireLab using docker-compose command.

  • Source code of the GrimoireLab components is available in src. Each directory is a Git submodule, so its contents will not be available after cloning the repository. To fetch all the data, and get the latest version, you can run the following command:

git submodule update --init --remote
  • How releases of GrimoireLab are built and tested: Building

Citation

If you use GrimoireLab in your research papers, please refer to GrimoireLab: A toolset for software development analytics:

APA style:

Dueñas S, Cosentino V, Gonzalez-Barahona JM, del Castillo San Felix A, Izquierdo-Cortazar D, Cañas-Díaz L, Pérez García-Plaza A. 2021. GrimoireLab: A toolset for software development analytics. PeerJ Computer Science 7:e601 https://doi.org/10.7717/peerj-cs.601

BibTeX / BibLaTeX:

@Article{duenas2021:grimoirelab,
  author = 	 {Dueñas, Santiago and Cosentino, Valerio and Gonzalez-Barahona, Jesus M. and del Castillo San Felix, Alvaro and Izquierdo-Cortazar,  Daniel and Cañas-Díaz, Luis and Pérez García-Plaza, Alberto},
  title = 	 {GrimoireLab: A toolset for software development analytics},
  journaltitle = {PeerJ Computer Science},
  date = 	 {2021-07-09},
  volume = 	 7,
  number = 	 {e601},
  doi = 	 {10.7717/peerj-cs.601},
  url = 	 {https://doi.org/10.7717/peerj-cs.601}}

Contributing

Contributions are welcome, please check the Contributing Guidelines.

Frequently asked questions

Is grimoirelab free to use?

grimoirelab is open source under the GPL-3.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does grimoirelab do?

GrimoireLab: platform for software development analytics and insights

What is grimoirelab written in?

grimoirelab is primarily written in Shell. Its source is publicly available at https://github.com/chaoss/grimoirelab, and it has 639 GitHub stars.