Single jenkins job for all repositories in a Github organization

jenkins github organization credentials
github branch source plugin example
jenkins github organization folder
jenkins job repository
jenkins poll scm multiple repositories
how to integrate jenkins with github
jenkins pipeline multiple repositories
jenkins build multiple branches

We own a Github organization with hundreds repositories that are authored by contributors. We would like to setup a Jenkins server that performs certain standard tasks for each commit in any of the repos in our Github organization. The intended CI flow is pretty simple:

  1. User commits a change to repo myorg/foobar
  2. Github organization-wide webhook for myorg calls the Jenkins server
  3. Jenkins runs a docker command to perform tasks for myorg/foobar
  4. Jenkins sets commit status to pending including link to command progress output
  5. Upon completion, Jenkins updates final commit status to success or failure

I am new to Jenkins and completely lost on which Plugins or Job type I need to set this up.

I tried creating a Jenkins "GitHub Organization" for my Github org, but it just tells me "This folder is empty, there are no repositories found that contain buildable projects". It's also unclear to me where the github organization webhook has to be configured.

We don't want to setup separate jobs/jenkinsfiles/webhook for all repos, but simply use a standard script that gets run for any commit in each repo, and trigger this via a single gh organization webhook. Is this possible?

Creating projects based on GitHub repository structure, Import all or a subset of repositories as jobs into the workspace from a GitHub user or Import a single repository's branches as jobs from a GitHub user or organization To define a job, you create a Pipeline script in a Jenkinsfile in the root� Github Organization Job Github organization, like a multi-branch pipeline, is one of the Jenkins project types. Mullitbranch pipleine can only configure pipelines for a single Git repository. Whereas a Jenins Github organization project can automatically configure multi-branch pipelines for all the repos in a Github organization.

Answering my own question:

As several people had pointed out, Jenkins assumes one job per repository. The Github Organization plugin didn't work well because it is clumsy and requires you commit and maintain a Jenkinsfile to each of your repos, which is specifically what I wanted to avoid.

The critical piece of information that I unaware of is that Jenkins has an excellent CLI and REST api to control jobs, and a single job configuration can easily be exported as a simple xml file.

So what I did is setup a the Jenkins job for one of the repos via the Jenkins GUI. Then I wrote a simple REST client that downloads the config.xml for this jobs, and creates or updates the Jenkins jobs for each of the repositories in our Github organization.

The builds automatically get triggered by Github (organization-wide) webhook if the URL matches that of any of the repositories. No special Github Organization plugin is needed.

Setup Jenkins Multibranch Pipeline and Organization, This pipeline uses the Jenkins Job DSL plugin to create the multi-branch for the repositories in organization and creates the Multi-Branch Pipelines in Jenkins . But this one should definitely be an exception of the rule -1 for all of them. 2. Create a Jenkins Job by clicking on New Item > Bitbucket Team/Project (you need Bitbucket Branch Source Plugin for this). Put your bitbucket owner and credentials under Projects. Under Local File insert "pom.xml", so that Jenkins can recognize to trigger the pipeline for every repository that has a pom. After doing so, your job will be able to scan automatically all your projects.

You have more than one requirement here. Let's go through one by one.

a) Jenkins GitHub Organization: This will scan all your GitHub organization, and create as many jobs as are needed to build your repositories because having just one job on Jenkins is not the standard. Basically, you lost history data (Jenkins has no idea it is building different stuff at every iteration). It says on the help "Scans a GitHub organization (or user account) for all repositories matching some defined markers.".

b) Try to see Jenkins as an automator, not something which will host all build/deploy logic. What I do is to create files like "build.sh", "deploy.sh", and so on. This way, I can build and deploy directly from my shell. So only after that I create scripts for Jenkins, that just call build/deploy scripts, no matter what they actually do. Jenkins doesn't need to know. A side effect is that all your projects "can be built the same way", no matter if they are NodeJS, Python, or whatever. Of course, you might need extra dependencies in some cases, and Docker can really help here.

c) I did something similar in the past, having less jobs than repositories/branches/pull-requests. Jenkins is kind of dump, and a few plugins can help here. But in your case, if you really want to have one job, you only need a regular parametrized job. The trick is that your Github Organization global webhook will not point to Jenkins. It needs to point to somewhere else, some code you maintain. This code can parse the Github payload, analyze it, can eventually calls back GitHub ("is there a pull request for this branch? no? then forget it") to enhance its decision tree, and at the end, trigger that single job on Jenkins with all parameters you were able to capture. Such parameters will tell the single job which repo to clone, env to deploy to, and that is it. You already know scripts names, since they are standard.

d) That said, I would ask... do you need a Jenkins? Can this parser-little-software actually clone your repo, and run a few scripts inside a Docker container? A builder-docker-container which has every dependency inside?

e) About "talking back" to GitHub, I did that using Python. There are GitHub libraries so I was able to get stuff back from Jenkins, and do API posts to feed GitHub with build statuses. Since I was actually using a Jenkins instance, my tool was a man-in-the-middle-broker. In your case, for a single job, a Docker container will play the role nicely.

Hopes this helps with a different perspective.

If you actually want to use a Jenkins instance, most of what I said here can still be used.

Jenkins pipeline to read the repository list for Github Organization , I have a very simple Jenkins build which is needed for several repositories. All these repositories have the same organisation, the same branches, etc. There is � A Jenkins plugin for analyzing the historical console output of a Job with the goal of determining which steps are taking the most time. jenkins-plugin timestamper-plugin Groovy MIT 5 22 5 1 Updated Oct 9, 2019

Assuming your Jenkins are running in a Linux or MacOS, or in Windows supporting Shell Script commands, configure a Jenkins job to execute the script below. Don't forget to replace the user and password fields and read the comment lines in order to understand and maybe improve it.

curl -i -u "user":"password" "https://github.com/your_organization" | grep "codeRepository" | awk -F '"' '{print $8}' | while read line; do
    mkdir "_temp_repo"
    cd "_temp_repo"

    # `--depth=1` clones the last commit only improving the speed of the clone
    # if you want to clone a specific branch add `-b branch` to the command below
    git clone --depth=1 "https://github.com"$line .

    # execute here your peding commands...

    git add .
    git commit -am "[pending] XPTO..."

    git push

    # execute here your success/failure commands...

    git add .
    git commit -am "[success/failure] XPTO..."

    git push

    cd ..
    rm -rfv "_temp_repo"
done

I would suggest create a SH file and execute in verbose mode: sh -x ./my_script.sh.

In order to perform it for every new update, setup a Github webhook to this job.

Reuse one Jenkins jobs for multiple Git (GitHub) repositories, This trigger only kicks git-plugin internal polling algo for every incoming With this, every new push to the repository automatically triggers a new build. Note that there's only one URL and it receives all post-receive POSTs for� Jenkins scans your entire GitHub organization and creates Pipeline jobs for each repository containing a Jenkinsfile—a text file that defines the process of building, testing, and deploying your project using Jenkins.

The standard approach would be to create a new multibranch pipeline which scans your organization for new repositories. Every repository should have a jenkinsfile with the instructions to build. But in general it is also possible to achieve what you are trying on a programmatical way.

What my approach would be:

  1. Create a Job Template as config.xml (shell script to run docker to check certain things)
  2. Scan GitHub to find new Repositories
  3. Create a new jenkins job based on the temnplate (ideally just replace the SCM link to the new location) How-to-create-a-job-using-the-REST-API-and-cURL
  4. Run that job

I would use the Folders Plugin to create a folder for this type of jobs.

If that is what you are really trying to do I could elaborate further.

GitHub, Multibranch Pipeline: build multiple branches of a single repository automatically. Organization Folders: scan a GitHub Organization or Bitbucket Team to� Login to your daily GitHub account and create new private repository. Follow the steps that are displayed to connect your local Android project repository with GitHub remote server. When it’s done,

Pipeline as Code, Should be no problem applying it in an organization with a huge amount of repositories! "We will need a huge amount of jobs in Jenkins (releasing, testing , Do not start treating specific repos differently in the global build scripts. Instead Regularly finds all repos, perhaps via the Git service REST-API. Particularly, if your organization utilizes GitHub Organizations or Bitbucket Teams, any time a developer creates a new repository with a Jenkinsfile, Jenkins will automatically detect it and create a Multibranch Pipeline project for it. This alleviates the need for administrators or developers to manually create projects for these new repositories.

Managing 1000+ Git repositories in Jenkins with a breeze, of Jenkins jobs for multiple repositories by using a single Jenkinsfile. with Github or any other versioning control tools supported by Jenkins. Organization Folder; afterwards the pipeline will be called every time you� In the example of a GitHub Organization job, we're talking to a GitHub Organization. Jenkins scans each repository in that Organization, and per repository, scan each branch. For each branch that meets the criteria - by default the presence of a Jenkinsfile - it creates the Multibranch Pipeline job for that repository.

CI/CD Building a pipeline used by multiple repositories with Jenkins , Jenkins GIT Reference Repository blocker in GitHub Organization Pipeline, where I have multi repositories in one folder. as a GIT-HUB Organization Job where I have multiple repositories under the "Townsuite" folder. Import all or a subset of repositories as jobs into the workspace from a GitHub user or organization Import a single repository’s branches as jobs from a GitHub user or organization The GitHub Organization project scans the repository, importing the Pipeline jobs it identifies according to your criteria.

Comments
  • Could you elaborate if (a) is possible icw/ a freestyle job config (i.e. run a fixed script without committing a Jenkinsfile to each git repo)? I succeeded in defining a freestyle job for a single repo, or scan myorg for all repos, however it is unclear to me how to apply this job to all scanned repos.
  • If you scan for repos, something needs to be there, like Jenkinsfile files. On a freestyle job, you don't need to necessarily configure a repository. It is just a job, that can only print a hello world and then exit. Then, you can call a few scripts to actually perform the clone, build, and so on (single-job approach). I had one repo called "jenkins-scripts", which was a job just to clone its contents to Jenkins. Then since I had only a Jenkins master node, all my jobs (I had many) were referencing something like "$WORKSPACE/../../scripts-jenkins/build.sh", for instance.
  • But keep in mind that GitHub talks events, and Jenkins needs specific instructions to do what you need. That's why there are Jenkins plugins to interoperate with GitHub. What I described on c) is a WebService that receives the payload, and calls Jenkins on that particular freestyle job, with specific parameters.