Notes: Jenkins Overview

jenkins-overview

# Jenkins Management
---
# Using Jenkins
 ## Using Credentials
Jenkins can store the following types of credentials:

| Type | Details |
| --- | --- |
| Secret text | a token such as an API token (e.g. a GitHub personal access token) |
| Username and password | which could be handled as separate components or as a colon separated string in the format username:password (read more about this in Handling credentials) |
| Secret file | which is essentially secret content in a file |
| SSH Username with private key | an SSH public/private key pair |
| Certificate | a PKCS#12 certificate file and optional password |
| Docker Host Certificate Authentication|  credentials |

To maximize security, credentials configured in Jenkins are stored in an encrypted form on the controller Jenkins instance (encrypted by the Jenkins instance ID)

**Credential Scopes:**

- *Global* - if the credential/s to be added is/are for a Pipeline project/item. Choosing this option applies the scope of the credential/s to the Pipeline project/item "object" and all its descendant objects.

- *System* - if the credential/s to be added is/are for the Jenkins instance itself to interact with system administration functions, such as email authentication, agent connection, etc. Choosing this option applies the scope of the credential/s to a single object only.

**Credential Domains:**
Credential domains are intended to help select correct credentials for each services. A credential domain has a specification (example: only hostname myservice.example.com, only URLs with the https protocol, only port 8443). 

**Credential Provider**

A credential provider connects Jenkins to an external credential vault. A credential provider is an extension point.

 - Jenkins (System) Credentials Provider
 - User Credentials Provider
- BlueOcean Folder Credentials
- Folder Credentials Provider

**Credentials Store**

Credentials providers expose credentials to Jenkins through the credentials store extension point. A credentials store is associated with a specific context within Jenkins. It either only supports the global domain, or has full support for custom domains. A credentials store will support a defined list of credentials scopes.

Internal credential stores will be responsible for storing actual credentials. 

 Ref: https://www.jenkins.io/doc/book/using/using-credentials/
 
 ## Using Search Box
 Every page in Jenkins has a search box on its top right that lets you get to your destination quickly, without multiple clicks.
 
 For example, if you type "foo #53 console", you’ll be taken to the console output page of the "foo" job build #53. If you have "XYZ" view, just type "XYZ" to get to that view. The search box comes with auto-completion to assist you.
 
 The search box is also context sensitive. If you are already somewhere in the foo job (perhaps you are looking at another build), then you can just type "#53 console" instead of "foo #53 console".
 
 If you want the search box to be case insensitive, go to your profile configuration page (/jenkins/user/\<your profile\>/configure) and activate the case insensitive search option.

![4df7ced6c3e8d51825007ac5a5c07494.png](:/9904842f5df54dc09c7f33c66b534862)
 Ref: https://www.jenkins.io/doc/book/using/searchbox/
 
## Time Zone
If your Jenkins instance is running in a different location than your own (for example: the server is in NY but you are in LA), then the NY time zone will most probably be used. This may be quite annoying if you need to compare build dates.

To see the time zone currently set, go to `jenkins_server/systemInfo` and see the `user.timezone` system property.

## Executor Starvation
If you see a little black clock icon in the build queue as shown below, it is an indication that your job is sitting in the queue unnecessarily.

The tool tip of the job name link next to the clock icon should tell you exactly why it is not building, but the common symptoms are as follows:
    - **Agents are offline:** your build needs to run on a particular agent, but the agent is offline. Go to http://server/jenkins/computer/AGENTNAME to understand why, and bring it back online. Or better yet, use labels and do not tie builds to specific agents, so that a single offline agents will not prevent your builds from starving.
    - **Waiting for an available executor on an agent:** your build needs to run on a particular agent, but the agent is already fully busy building other things, and your build is waiting for "too long" compared to the time it takes to execute it — in other words, it does not make sense to wait for 5 minutes when the build itself finishes in 2 minutes. Use labels so that builds can run on any machine that satisfies the system requirements, and in this way you can add more agents to improve the turn-around time.
   - **Waiting for an available executor on a label:** all the agents that have the given label are fully busy doing other things. It is time to add more agents.


## Remote Access Api
Jenkins provides machine-consumable remote access API to its functionalities. Currently it comes in three flavors:
  - XML
  - JSON with JSONP support
  - Python

Remote API can be used to do things like these:
  - retrieve information from Jenkins for programmatic consumption.
  - trigger a new build
  - create/copy jobs


**Jobs without parameters**

You merely need to perform an HTTP POST on `JENKINS_URL/job/JOBNAME/build`.

**Jobs with parameters**

Simple example - sending "String Parameters":
```
curl JENKINS_URL/job/JOB_NAME/buildWithParameters \
  --user USER:TOKEN \
  --data id=123 --data verbosity=high
```

 Sample script:
 ```
 #!/bin/bash
JENKINS_URL="http://jenkins-master.home.arpa:8080/"
JENKINS_USER=admin
JENKINS_PASSWORD=<JENKINS_TOKEN>


# Jenkins api
echo "==> Trigger jenkins job test"
curl -u "$JENKINS_USER:$JENKINS_PASSWORD" -X POST $JENKINS_URL/job/test/build

sleep 30
curl -u "$JENKINS_USER:$JENKINS_PASSWORD" -X POST $JENKINS_URL/job/test/api/json | jq -r 
 ```
 
 Ref: https://www.jenkins.io/doc/book/using/remote-access-api/
 
 ## Using Jenkins agents
 
The Jenkins architecture is designed for distributed build environments. It allows us to use different environments for each build project balancing the workload among multiple agents running jobs in parallel.  The Jenkins controller is the original node in the Jenkins installation. 

The Jenkins controller administers the Jenkins agents and orchestrates their work, including scheduling jobs on agents and monitoring agents. Agents may be connected to the Jenkins controller using either local or cloud computers.

**TYPES: https://www.jenkins.io/doc/book/using/using-agents/**

Delegating the first job to agent1
  - Go to your jenkins dashboard;
  - Select New Item on side menu;
  - Enter a name. (e.g.: First Job to Agent1)
  - Select the Freestyle project and press OK;
  - Check the option: `Restrict where this project can be run;`
  - Fill the field: label with the agent1 label; (e.g.: agent1)

Ref: https://www.jenkins.io/doc/book/using/using-agents/
 
---
# Managing Jenkins
## Configuring the System

**Home directory:**
Jenkins needs some disk space to perform builds and keep archives. One can check this location from the configuration screen of Jenkins. By default, this is set to ~/.jenkins, and this location will initially be stored within your user profile location

**# of executors:**
This refers to the total number of concurrent job executions that can take place on the Jenkins machine. This can be changed based on requirements. Sometimes the recommendation is to keep this number the same as the number of CPU on the machines for better performance.

**Environment Variables:**
This is used to add custom environment variables which will apply to all the jobs. These are `key-value` pairs and can be accessed and used in Builds wherever required.

**Jenkins URL:**
By default, the Jenkins URL points to `localhost`. If you have a domain name setup for your machine, set this to the domain name else overwrite localhost with IP of machine. This will help in setting up slaves and while sending out links using the email as you can directly access the Jenkins URL using the environment variable `JENKINS_URL` which can be accessed as `${JENKINS_URL}`.

**Email Notification:**
In the email Notification area, you can configure the SMTP settings for sending out emails. This is required for Jenkins to connect to the SMTP mail server and send out emails to the recipient list.

Under E-mail Notification, put the mentioned details below:
-     SMTP server → smtp.gmail.com
-     Select Use SMTP Authentication
-     Put your gmail id
-     Put your gmail password
-     Use SSL select
-     SMTP port 465
-     Save the configuration


## Managing Tools

## Managing Plugins
Here you can install a wide or different variety of third-party plugins from different Source code management tools such as Git, ClearCase or Mercurial, to code quality and code coverage metrics reporting. We can download, install, update, or remove the plugins from the Manage Plugins screen.

## About Jenkins
This option will show the version and license information of the Jenkins you are running. As well as it displays the list of all third party libraries.

## System Information
This option displays a list of all the current Java system properties and system environment variables. Here you can check what version of Java is currently running in, what user it is running under, and so forth.

Ref: 

## Managing Nodes
**Agent Communication: **

There are different ways to establish a connection between controller and agent:

- *The SSH connector:* Configuring an agent to use the SSH connector is the preferred and the most stable way to establish controller-agent communication. Jenkins has a built-in SSH client implementation. This means that the Jenkins controller can easily communicate with any machine with an SSH server installed. The only requirement is that the public key of the controller is part of the set of the authorized keys on the agent. Once the host and SSH key is defined for a new agent, Jenkins will establish a connection to the machine and bootstrap the agent process.

- *The inbound connector:* In this case the communication is established starting the agent through a connection initiated by an agent program. With this connector the agent is launched in the machine in 2 different ways:
       1. *Manually:* by navigating to the Jenkins controller URL in a browser on the agent. Once the Java Web Start icon is clicked, the agent will be launched on the machine. The downside of this approach is that the agents cannot be centrally managed by the Jenkins controller and each/stop/start/update of the agent needs to be executed manually on the agent’s machine in versions of Jenkins older than 1.611. This approach is convenient when the controller cannot instantiate the connection with the client, for example: with agents running inside a firewalled network connecting to a controller located outside the firewall.
       2. *As a service:* First you’ll need to manually launch the agent using the above method. After manually launching it, jenkins-slave.exe and jenkins-slave.xml will be created in the agent’s work directory.

Ref: https://www.jenkins.io/doc/book/managing/nodes/
Ref: https://www.jenkins.io/doc/book/scaling/architecting-for-scale/#distributed-builds-architecture
Ref: https://www.baeldung.com/ops/jenkins-slave-node-setup

## Managing Security
Configure Global Security option provides the ability to set up users and their relevant permissions on the Jenkins instance. By default, you will not want everyone to be able to define builds or other administrative tasks in Jenkins. So Jenkins provides the ability to have a security configuration in place.

Ref: https://www.jenkins.io/doc/book/security/managing-security/

## Controller Isolation
Ref: https://www.jenkins.io/doc/book/security/controller-isolation/

---
# System Administrator
## Types of installation
- Docker
- Kubernetes
- Linux
- WAR files
- Windows

Ref: https://www.jenkins.io/doc/book/installing/

## Initial settings
Jenkins initialization can also be controlled by run time parameters passed as arguments. Command line arguments can adjust networking, security, monitoring, and other settings.
| Command Line Parameter	| Description |
| --- | --- |
| --httpPort=$HTTP_PORT |	Runs Jenkins listener on port $HTTP_PORT using standard http protocol. The default is port 8080. |
| --httpListenAddress=$HTTP_HOST |	Binds Jenkins to the IP address represented by $HTTP_HOST. The default is 0.0.0.0 — i.e. listening on all available interfaces. For example, to only listen for requests from localhost, you could use: --httpListenAddress=127.0.0.1 |
| --httpsPort=$HTTPS_PORT	| Uses HTTPS protocol on port $HTTPS_PORT. This option does not impact the root URL being generated within Jenkins logic (UI, inbound agent files, etc.). It is defined by the Jenkins URL specified in the global configuration. |
| --httpsListenAddress=$HTTPS_HOST	| Binds Jenkins to listen for HTTPS requests on the IP address represented by $HTTPS_HOST. |
| --prefix=$PREFIX	| Runs Jenkins to include the $PREFIX at the end of the URL. For example, set --prefix=/jenkins to make Jenkins accessible at http://myServer:8080/jenkins |

Some Jenkins behaviors are configured with Java properties. Java properties are set from the command line that started Jenkins. Property assignments use the form -DsomeName=someValue to assign the value someValue to the property named someName. For example, to assign the value true to a property testName, the command line argument would be -DtestName=true.

Ref: https://www.jenkins.io/doc/book/installing/initial-settings/
Ref: https://www.jenkins.io/doc/book/managing/system-properties/#Featurescontrolledbysystemproperties-PropertiesinJenkinsCore

## Managing systemd services
Linux package installers use `systemd` to manage services. 

**Viewing service configurations*:*
`systemctl cat jenkins`

**Overriding service configurations**

- Debian:    /lib/systemd/system/jenkins.service
- Red Hat:  /usr/lib/systemd/system/jenkins.service
- openSUSE:  /usr/lib/systemd/system/jenkins.service

The main service unit is read-only and not intended to be edited manually. Values may be overridden in the drop-in unit (override.conf file) for the service. Edit the drop-in unit with:
`systemctl edit jenkins`

The override.conf file is stored at /etc/systemd/system/jenkins.service.d/override.conf and can be used to customize the service. Note that such customizations must be done in a [Service] section in order to take effect. Example content of the override.conf file might include:

| Action | Command |
| --- | --- |
| starting service | systemctl start jenkins |
| Stopping service | systemctl stop jenkins |
| Restarting service | systemctl restart jenkins |
| Reloading service definitions | systemctl daemon-reload |
| Reading service logs | journalctl -u jenkins |
| Viewing service status | systemctl status jenkins |

Ref: https://www.jenkins.io/doc/book/system-administration/systemd-services/index.html

## Viewing logs
### Logs on the system
When running jenkins.war manually with java -jar jenkins.war, all logging information by default is output to standard out. Many Jenkins native packages modify this behavior to ensure logging information is output in a more conventional location for the platform.

#### Linux
By default logs can be viewed by running journalctl -u jenkins.service. To customize the log location, run `systemctl edit jenkins` and add the following:

```
[Service]
Environment="JENKINS_LOG=%L/jenkins/jenkins.log"
```

### Windows
By default logs should be at `%JENKINS_HOME%/jenkins.out `and `%JENKINS_HOME%/jenkins.err,` unless customized in `%JENKINS_HOME%/jenkins.xml`

Ref: https://www.jenkins.io/doc/book/system-administration/viewing-logs/

 ## Backing-up/Restoring Jenkins

### Back up the Controller Key Separately

Never include the controller key in your Jenkins backup!

The controller key is used to encrypt data in the secrets directory that secures credentials. It is stored in the `$JENKINS_HOME/secrets/hudson.util`. Secret file in the `$JENKINS_HOME/secrets/` directory and encrypted with `master.key`. If you need to restore a system from a backup, you will need this file. And, if someone else accesses your backups and has this key, they have full access to all your information.

You should *treat your controller key like you treat your SSH private key and NEVER include it in a regular backup*. Instead, back up the `master.key` file separately and store it in a very secure location away from your other backups. It is a very file small that is seldom changed. If you need to do a full system restore, you will need to restore the rest of the system and then apply the backup of the master.key file separately.

### Which Files Should Be Backed Up?
The number of files you back up can affect both the time required to run the backup and the size of the resulting backup. It also impacts the complexity of restoring the system from the backup. Here we discuss why various files should be backed up and list some files that could safely be excluded from at least some backups.

Backing up the entire $JENKINS_HOME directory preserves the entire Jenkins instance. To restore the system, just copy the entire backup to the new system.

Note, however, that JENKINS_HOME includes a number of files that do not really need to be backed up. Selecting specific directories and files to back up yields smaller backups but may require a greater effort to restore a system. One approach is to back up different directories on different schedules.

Configuration files are stored directly in the *$JENKINS_HOME directory. ./config.xml is the main Jenkins configuration file*. Other configuration files also have the `.xml suffix`. Specify `$JENKINS_HOME/*.xml to back up all configuration files.
Configuration files can also be stored in an SCM repository. This keeps copies of all previous versions of each file that can be retrieved using standard SCM facilities.

The `$JENKINS_HOME/jobs` directory contains information related to all the jobs you create in Jenkins.
 - ./builds — Contains build records
 - ./builds/archive — Contains archived artifacts
   - Back this up if it is important to retain these artifacts long-term
   -  These can be very large and may make your backups very large
 - ./workspace — Contains files checked out from the SCM
    - It is usually not necessary to back up these files. You can perform a clean checkout after restoring the system.
 - ./plugins/*.hpi — Plugin packages with specific versions used on your system
 - ./plugins/*.jpi — Plugin packages with specific versions used on your system

 Ref: https://www.jenkins.io/doc/book/system-administration/backing-up/
 
## Reverse proxy

Ref: https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-with-jenkins/index.html