While working on our {gitdown} package, I wanted to show more information on the report generated by retrieving all issues of my Gitlab / Github repositories. Indeed, titles of issues could improve readability and information of the git report of this package. How can I download all issues information from git repositories?
Context of {gitdown} development
Table of Contents
{gitdown} is a package to create a HTML report from your git commits. This retrieves issue numbers and other patterns listed in your commit messages to present all commits linked to each pattern. We built this package for a client in Pharma who wanted to link all our commits to their own project management system. Indeed, in Pharma, every modification of the software built needs to be documented. Here, in each of our commits messages, we added a reference to their project management system, which was used in parallel of our Gitlab development workflow.
Let’s go back to the topic of this blog post. Here, I want to download the list of issues on my Gitlab/Github account to add information in my {gitdown} book. With this list, I’ll be able to add the title of the issue instead of #1 only, as in the figure above.
Create a fake project
For tests and examples in the {gitdown} package, function fake_repo() build a fake project with some files and commits.
# remotes::install_github("ThinkR-open/gitdown")
library(gitdown)
# Create the repo here
repo <- fake_repo(path = "repo.rtask", as.package = TRUE)
# Content of the repo
fs::dir_tree(repo)## repo.rtask
## ├── NEWS.md
## ├── R
## │ └── my_mean.R
## ├── example.txt
## └── vignettesThis repository contains specific git messages. You may already be using the issue number to link to your commit messages, such as #123 for a link to the issue 123. Here we have also added specific text patterns to link to a client project management system, such as ticket123 for their internal ticket named 123. You can see these patterns in the next commit.
library(git2r)
summary(last_commit(repo))## Commit: 0fb0129f3397c13c13dad76e91e80719405a48b9
## Author: Alice <[email protected]>
## When: 2020-08-10 15:33:45 GMT
##
## Add NEWS
##
## issue #32.
## issue #1.
## issue#12
## ticket6789.
## ticket1234
## Creation of the NEWS file for version 0.1.
## 1 file changed, 3 insertions, 0 deletions
## NEWS.md | -0 +3 in 1 hunkIn my previous blog post “Transform a folder as git project synchronized on Github or Gitlab”, I already created two distant repositories of my project on Gitlab (https://gitlab.com/statnmap/repo.rtask) and Github (https://github.com/statnmap/repo.rtask).
In both these repositories, there are two issues.
Download issues from Gitlab
remotes::install_github("statnmap/gitlabr")library(gitlabr)
library(dplyr)Connection to Gitlab from R
You will need to create your access token. You can generate one in the web interface under “
# Store your token in .Renviron and restart your session
usethis::edit_r_environ()
# Add: GITLAB_COM_TOKEN=YourTokenHere
# You can verify it worked
Sys.getenv("GITLAB_COM_TOKEN")Then you can connect to your project on Gitlab. Object my_project() created is a function allowing communication with the Gitlab API, provided that you use the correct functions.
I highly recommend that you use the project ID, otherwise the function will try to download the entire gitlab.com to search for your own project. You can find your project ID on Gitlab in the General Settings.
# Connect as a fixed user to a gitlab instance
my_project <- gl_project_connection(
gitlab_url = "https://gitlab.com",
project = 20384533, #repo.rtask",
private_token = Sys.getenv("GITLAB_COM_TOKEN")
)Download Gitlab issues of a specific project
We use gl_list_issues to retrieve the list of issues, using our connection. We retrieve a table with a row for each issue.
my_project_issues <- my_project(gl_list_issues)
my_project_issues## # A tibble: 2 x 32
## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 6952… 2 20384533 A se… The blog p… open… 2020-08-0… 2020-08-0… 4809823 Sébastien …
## 2 6952… 1 20384533 An e… No desc in… open… 2020-08-0… 2020-08-0… 4809823 Sébastien …
## # merge_requests_count <chr>, upvotes <chr>, downvotes <chr>, confidential <chr>,
## # web_url <chr>, time_stats.time_estimate <chr>, time_stats.total_time_spent <chr>,
## # task_completion_status.count <chr>, task_completion_status.completed_count <chr>,
## # has_tasks <chr>, `_links.self` <chr>, `_links.notes` <chr>, `_links.award_emoji` <chr>,
## # `_links.project` <chr>, references.short <chr>, references.relative <chr>,
## # references.full <chr>Download issues from Github
I use {gh} to download issues from Github, with the notes provided by Jenny Brian “How to obtain a bunch of GitHub issues or pull requests with R”. First, you will need to set your Github Token as environment variable (See ?gh_token for more information).
library(gh)
library(purrr)
# gh_token()
issue_list <- gh(
"/repos/:owner/:repo/issues",
owner = "statnmap",
repo = "repo.rtask",
state = "all", since = "2015-09-01T00:00:00Z",
.limit = Inf)
issue_df <- issue_list %>%
tibble(number = map_int(., "number"),
id = map_int(., "id"),
title = map_chr(., "title"),
state = map_chr(., "state"),
n_comments = map_int(., "comments"),
opener = map_chr(., c("user", "login")),
created_at = map_chr(., "created_at") %>% as.Date())
issue_df## # A tibble: 2 x 8
## . number id title state n_comments opener created_at
## 1 <named list… 2 6.75e8 A second issue to illust… open 0 statnm… 2020-08-06
## 2 <named list… 1 6.75e8 An example of issue open 0 statnm… 2020-08-06Create a git report that uses issues names with {gitdown}
Here comes {gitdown} to create the report. The standard report that takes informations in the commit messages is the following one.
git_down(repo = repo)
As I downloaded tables of my issues with their names from Github and Gitlab, I can use part of the table to improve the {gitdown} report. I use parameter pattern.table which requires a two-column table of pattern and title. Pattern must be written in full, like #1 for issue 1.
# From gitlab
my_pattern_table <- my_project_issues %>%
mutate(
pattern = paste0("#", iid),
title = paste(pattern, title)
) %>%
select(pattern, title)
# From github
my_pattern_table <- issue_df %>%
mutate(
pattern = paste0("#", number),
title = paste(pattern, title)
) %>%
select(pattern, title)
my_pattern_table## # A tibble: 2 x 2
## pattern title
## <chr> <chr>
## 1 #2 #2 A second issue to illustrate a blog post
## 2 #1 #1 An example of issue# Launch gitdown
git_down(repo = repo, pattern.table = my_pattern_table)
Create a git report with user specific patterns
You will probably tell me that the report with issues, you can have it online with your Gitlab / Github account. That’s not false… The report is dedicated to people who don’t know what is a git repository. It is also meant to be delivered for on offline documentation for a specific release of your software.
Also, this report can be used in combination with an external project management system, the one of your client for instance. To do so, you will need to define a text pattern to identify the tickets of your clients and write this pattern when relevant in your commits messages.
In the fake_repo() example, there is indeed a second pattern that we identified with ticket1234 in commit messages. You saw it in the last commit for instance.
summary(last_commit(repo))## Commit: 0fb0129f3397c13c13dad76e91e80719405a48b9
## Author: Alice <[email protected]>
## When: 2020-08-10 15:33:45 GMT
##
## Add NEWS
##
## issue #32.
## issue #1.
## issue#12
## ticket6789.
## ticket1234
## Creation of the NEWS file for version 0.1.
## 1 file changed, 3 insertions, 0 deletions
## NEWS.md | -0 +3 in 1 hunkHence, we can create the report for this specific pattern with a pattern.table too.
pattern.table <- data.frame(
pattern = c("ticket1234", "ticket6789"),
title = c("Design of the Shiny app",
"Core of the app in Rmd")
)
git_down(repo, pattern = c("Tickets" = "ticket[[:digit:]]+"),
pattern.table = pattern.table)
For the same reasons, we built {testdown}, a package to turn your {testthat} results into an HTML gitdown report.
Try it yourself and tell us!
Now that you saw the possibilities of {gitdown}, {gitlabr} and {gh}, let us know what you’ll do with those!
On {gitdown}, if you find any problem, if you see features to add, you can open an issue or a pull request on the Github repository: https://github.com/ThinkR-open/gitdown
Note: {gitlabr} was originally developed by Jirka Lewandowski (https://github.com/jirkalewandowski/gitlabr) who sadly passed away last year. So I took over the development of this package on my own Github account. As it hasn’t been updated for some time, it doesn’t work completely correctly with the latest versions of {tibble}, {dplyr}, … So I started to make some changes in the “master branch“. If you want to help updating this package, come and contribute to the Github project : https://github.com/ThinkR-open/gitlabr, propose “pull requests” and open “issues”.
