Jenkins

What is Jenkins?

Jenkins is an open source automation server. It helps automate the parts of software development related to building, testing, and deploying, facilitating continuous integration, and continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat

Your Jenkins Belongs to Us Now: Abusing Continuous Integration Systemscrowdstrike.com

File Location:

Not a lot is recorded with default Jenkins logs.

• Docker Logs:

  • var/snap/docker/common/var-lib-docker/volumes/13e15a1c49a2cccb9acf4dc096b7bd380de8bba800709ce7d15a92b5f23dfb97/_data/log

• Log Recorders:

  • /var/jenkins_home/logs

• Default Locations:

  • /var/jenkins_home/

  • /var/jenkins_home/config.xml

  • /var/jenkins_home/credentials.xml

  • /var/jenkins_home/jobs/JOBNAME

Config.xml File:

The config file will tell you everything you need to know about a Jenkins instance:

1. If anonymous access it turned on

2. What type of user access control is enabled

   1. Every user can do anything (including anonymous users).

   2. Only logged in user's have full access to everything.

   3. Only certain users can perform certain tasks (role-based and matrix based access).

3. What type of authentication is allowed:

   1. LDAP

   2. Active Directory

   3. Jenkins' own user database

   4. Unix users/groups

4. Where the Workspace and Builds directory are located.

<?xml version='1.1' encoding='UTF-8'?>
<hudson>
  <disabledAdministrativeMonitors>
    <string>hudson.util.DoubleLaunchChecker</string>
  </disabledAdministrativeMonitors>
  <version>2.452.2</version>
  <numExecutors>2</numExecutors>
  <mode>NORMAL</mode>
  <useSecurity>true</useSecurity> < Disabling this will turn off authentication
  <authorizationStrategy class="hudson.security.GlobalMatrixAuthorizationStrategy">  < User access control method.
    <permission>USER:hudson.model.Hudson.Administer:admin</permission>
    <permission>USER:hudson.model.Hudson.Read:excitedcactus</permission>
    <permission>USER:hudson.model.Item.Configure:excitedcactus</permission>
    <permission>USER:hudson.model.Item.Create:excitedcactus</permission>
  </authorizationStrategy>
  <securityRealm class="hudson.security.HudsonPrivateSecurityRealm"> < Allow authentication for users in Jenkins' own user database.
    <disableSignup>true</disableSignup>
    <enableCaptcha>false</enableCaptcha>
  </securityRealm>
  <disableRememberMe>false</disableRememberMe>
  <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
  <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}</workspaceDir>
  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
  <markupFormatter class="hudson.markup.EscapedMarkupFormatter"/>
  <jdks/>
  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
  <clouds/>
  <quietPeriod>5</quietPeriod>
  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
  <views>
    <hudson.model.AllView>
      <owner class="hudson" reference="../../.."/>
      <name>all</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
    </hudson.model.AllView>
  </views>
  <primaryView>all</primaryView>
  <slaveAgentPort>50000</slaveAgentPort>
  <label></label>
  <crumbIssuer class="hudson.security.csrf.DefaultCrumbIssuer">
    <excludeClientIPFromCrumb>false</excludeClientIPFromCrumb>
  </crumbIssuer>
  <nodeProperties/>
  <globalNodeProperties/>
  <nodeRenameMigrationNeeded>false</nodeRenameMigrationNeeded>
</hudson>

AuthorizationStrategy:

• Authorization Strategy, which determines who has access to what.

  • Unsecured

  • FullControlOnceLoggedInAuthorizationStrategy

  • LegacyAuthorizationStrategy

<authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>

UseSecurity:

If this is disabled, Jenkins will allow authentication without providing credentials. This will allow any person to access Jenkins with full access.

<useSecurity>false</useSecurity>

SecurityRealm:

Security Realm, which determines users and their passwords, as well as what groups the users belong to. In the below snippet, Active Directory users are allowed to access Jenkins.

<securityRealm class="hudson.security.SecurityRealm$ActiveDirectory"/>

Credentials.xml

• File requires root access to read.

• Jenkins service account is created on Jenkins install and does not have root access typically.

• To decrypt passwords in Script Console, the values in the <password> fields must be provided.

<?xml version='1.1' encoding='UTF-8'?>
<com.cloudbees.plugins.credentials.SystemCredentialsProvider plugin="credentials@1355.v46f52a_b_98d64">
  <domainCredentialsMap class="hudson.util.CopyOnWriteMap$Hash">
    <entry>
      <com.cloudbees.plugins.credentials.domains.Domain>
        <specifications/>
      </com.cloudbees.plugins.credentials.domains.Domain>
      <java.util.concurrent.CopyOnWriteArrayList>
        <com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
          <scope>GLOBAL</scope>
          <id>fcdec1d6-ffc1-4401-80f9-f6af5f53095e</id>
          <description></description>
          <username>admin</username>
          <password>{AQAAABAAAAAQtH1ZOWh9FNBwjn4jm+ZhNDu6GPGPhB3Q/Ws/bHmsW/I=}</password>
          <usernameSecret>false</usernameSecret>
        </com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
        <com.dabsquared.gitlabjenkins.connection.GitLabApiTokenImpl plugin="gitlab-plugin@1.8.1">
          <scope>GLOBAL</scope>
          <id>asd</id>
          <description></description>
          <apiToken>{AQAAABAAAAAwhMi29OfNpWa3x+J6URTcSpy4xLXqojQwEaXgoSJUd9CIcCvEE7NGHTg==}</apiToken>
        </com.dabsquared.gitlabjenkins.connection.GitLabApiTokenImpl>
      </java.util.concurrent.CopyOnWriteArrayList>
    </entry>
  </domainCredentialsMap>
</com.cloudbees.plugins.credentials.SystemCredentialsProvider>

Groovy Script Console:

Script console can execute Groovy based scripts and run commands on the OS. This can be leveraged for post compromise actions like lateral movement, credential theft, execution, and persistence.

• Outputs or commands are not logged by default.

Permissions:

• Access is controlled by the Overall Administrator permission in Jenkins.

Post Exploitation:

Any normal type of post exploitation action that you can think of can be performed through the Jenkins Script Console.

1. Persistence

   1. Setup scheduled tasks or services (Depends on OS).

2. Lateral Movement

   1. Drop a beacon or establish a connection with a C2 and then move deeper into network.

   2. Decrypt credentials and move into cloud/GitHub environment.

3. Execution of any command available.

   1. Recon commands.

   2. Creating users.

   3. Adding users to groups.

4. Credential Dumping:

   1. Executing Mimikatz.

   2. Reading files that contain plaintext passwords.

   3. Decrypting Jenkins credentials.

new File('/var/jenkins_home/config.xml').text
new File('/etc/shadow').text

Credential Theft:

Users can read stored Jenkins credentials and decrypt them from the Script Console.

There are many types of credentials stored in Jenkins:

1. GitHub Tokens.

2. SSH keys and usernames/passwords.

3. Active Directory usernames/passwords.

4. Cloud credentials.

To decrypt the credentials, the user must provide the encrypted password found in credentials.xml in the <password> field.

new File('/var/jenkins_home/credentials.xml').text

OUTPUT: <password>{AQAAABAAAAAQtH1ZOWh9FNBwjn4jm+ZhNDu6GPGPhB3Q/Ws/bHmsW/I=}</password>

println(hudson.util.Secret.decrypt("{AQAAABAAAAAQtH1ZOWh9FNBwjn4jm+ZhNDu6GPGPhB3Q/Ws/bHmsW/I=}"))

Script ConsoleScript Console

Process Genealogy:

Whenever a user executes commands from Script Console, it should be spawning from a Java process. This makes hunting for post exploitation activities from a threat actor detectable.

Suspicious processes should be spawning from:

• Linux:

  • usr/bin/java

• Windows:

  • java.exe

Jobs:

If Script Console is not accessible, jobs can instead be used to execute code on the underlying OS.

Jobs can be created to execute shell commands or a Windows batch commands (depending on OS). In the event that the user's permissions don't allow them to create a job, existing jobs may also be reconfigured as a means of executing shell commands.

Jobs Execution Artifacts:

In the Linux version of Jenkins, build instructions will be written and executed from the /tmp folder with the Jenkins naming convention. These files are immediately deleted after execution, so carving for them will be your best shot at finding them.

• /tmp/jenkins17741847249180715264.sh

There will be no artifacts related to the shell commands in the workspaces directory.

• /var/jenkins_home/workspace/test

Job Config Artifacts:

The config.xml file for jobs can be used to find any shell commands that the TA might've added.

• /var/jenkins_home/jobs/test/config.xml

<?xml version='1.1' encoding='UTF-8'?>
<project>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties>
    <com.dabsquared.gitlabjenkins.connection.GitLabConnectionProperty plugin="gitlab-plugin@1.8.1">
      <gitLabConnection></gitLabConnection>
      <jobCredentialId></jobCredentialId>
      <useAlternativeCredential>false</useAlternativeCredential>
    </com.dabsquared.gitlabjenkins.connection.GitLabConnectionProperty>
    <hudson.plugins.throttleconcurrents.ThrottleJobProperty plugin="throttle-concurrents@2.14">
      <categories class="java.util.concurrent.CopyOnWriteArrayList"/>
      <throttleEnabled>false</throttleEnabled>
      <throttleOption>project</throttleOption>
      <limitOneJobWithMatchingParams>false</limitOneJobWithMatchingParams>
      <paramsToUseForLimit></paramsToUseForLimit>
    </hudson.plugins.throttleconcurrents.ThrottleJobProperty>
  </properties>
  <scm class="hudson.scm.NullSCM"/>
  <canRoam>true</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers/>
  <concurrentBuild>false</concurrentBuild>
  <builders>
    <hudson.tasks.Shell>                               < Initial startup command
      <command>whoami</command>
      <configuredLocalRules/>
    </hudson.tasks.Shell>
  </builders>
  <publishers/>
  <buildWrappers/>
</project>

Build Version Artifacts:

If anonymous access is enabled, the user who created the Freestyle project or initiated the build will not be known.

• /var/jenkins_home/jobs/test/builds/3/build.xml

<?xml version='1.1' encoding='UTF-8'?>
<build>
  <actions>
    <hudson.model.CauseAction>
      <causeBag class="linked-hash-map">
        <entry>
          <hudson.model.Cause_-UserIdCause>
            <userId>admin</userId>                    < Who initiated build.
          </hudson.model.Cause_-UserIdCause>
          <int>1</int>
        </entry>
      </causeBag>
    </hudson.model.CauseAction>
    <jenkins.metrics.impl.TimeInQueueAction plugin="[e-mail]51df8df52ec">
      <queuingDurationMillis>20</queuingDurationMillis>
      <blockedDurationMillis>0</blockedDurationMillis>
      <buildableDurationMillis>16</buildableDurationMillis>
      <waitingDurationMillis>1</waitingDurationMillis>
    </jenkins.metrics.impl.TimeInQueueAction>
  </actions>
  <queueId>2</queueId>
  <timestamp>1720410156883</timestamp>
  <startTime>1720410156896</startTime>
  <result>SUCCESS</result>
  <duration>58</duration>
  <charset>UTF-8</charset>
  <keepLog>false</keepLog>
  <builtOn></builtOn>
  <workspace>/var/jenkins_home/workspace/test</workspace>
  <hudsonVersion>2.452.2</hudsonVersion>
  <scm class="hudson.scm.NullChangeLogParser"/>
  <culprits class="java.util.Collections$UnmodifiableSet">
    <c class="sorted-set"/>
  </culprits>
</build>

Build Log File:

The Script Console output will be stored in a log file after a successful build.

• /var/jenkins_home/jobs/<Build name>/builds/<Build #>/log

Build History:

Users with access to the build history can pilfer through these in order to find sensitive information:

1. GitHub Tokens

2. Secrets/passwords

UserContent:

Files in the UserContent directory are not subject to any access controls beyond requiring Overall/Read access.

This makes it a viable area for storing various things:

1. Script Console output.

2. Console/secrets can be redirected to this folder after each build.

3. Storing webshells.

Files can be accessed by navigating to the following URL:

• http://jenkins/userContent/file.txt

This directory should be checked to ensure nothing nefarious has been written to it.

• /var/jenkins_home/UserContent

• C:/Program Files (x86)/Jenkins/userContent

User ContentUser Content

Active Directory Plugin:

The Active Directory Plugin allows AD users to authenticate to Jenkins and also allows anonymous read access as an option.

Failed logins:

By default, failed authentications will be logged within the following locations:

• Jenkins logs

• 4625 Event ID

4625:

An account failed to log on.

Subject:
	Security ID:		SYSTEM
	Account Name:		WIN-9ITB345Q7UU$
	Account Domain:		MIKO
	Logon ID:		0x3E7

Logon Type:			3

Account For Which Logon Failed:
	Security ID:		NULL SID
	Account Name:		pekora
	Account Domain:		MIKO

Failure Information:
	Failure Reason:		Unknown user name or bad password.
	Status:			0xC000006D
	Sub Status:		0xC000006A

Process Information:
	Caller Process ID:	0x224
	Caller Process Name:	C:\Windows\System32\lsass.exe

Network Information:
	Workstation Name:	WIN-9ITB345Q7UU
	Source Network Address:	10.120.116.21
	Source Port:		47588

Detailed Authentication Information:
	Logon Process:		Advapi  
	Authentication Package:	MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
	Transited Services:	-
	Package Name (NTLM only):	-
	Key Length:		0

Successful logins:

By default, successful logins will ONLY be logged on the authenticating server (DC) and not in Jenkins.

In both successful and failed logins, the source IP is the Jenkins server and the workstation name is the authenticating server (DC).

The following event IDs will be logged on the DC for any successful authentication:

• 4624 An account was successfully logged on.

• 4628 A logon was attempted using explicit credentials.

• 4634 An account was logged off.

• 4627 Group membership information.

• 4776 The computer attempted to validate the credentials for an account.

4624:

An account was successfully logged on.

Subject:
	Security ID:		SYSTEM
	Account Name:		WIN-9ITB345Q7UU$
	Account Domain:		MIKO
	Logon ID:		0x3E7

Logon Information:
	Logon Type:		3
	Restricted Admin Mode:	-
	Virtual Account:		No
	Elevated Token:		Yes

Impersonation Level:		Impersonation

New Logon:
	Security ID:		MIKO\pekora
	Account Name:		pekora
	Account Domain:		MIKO
	Logon ID:		0x1EF8EE
	Linked Logon ID:		0x0
	Network Account Name:	-
	Network Account Domain:	-
	Logon GUID:		{00000000-0000-0000-0000-000000000000}

Process Information:
	Process ID:		0x224
	Process Name:		C:\Windows\System32\lsass.exe

Network Information:
	Workstation Name:	WIN-9ITB345Q7UU
	Source Network Address:	10.120.116.21
	Source Port:		47620

Detailed Authentication Information:
	Logon Process:		Advapi  
	Authentication Package:	MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
	Transited Services:	-
	Package Name (NTLM only):	-
	Key Length:		0

4628:

A logon was attempted using explicit credentials.

Subject:
	Security ID:		SYSTEM
	Account Name:		WIN-9ITB345Q7UU$
	Account Domain:		MIKO
	Logon ID:		0x3E7
	Logon GUID:		{00000000-0000-0000-0000-000000000000}

Account Whose Credentials Were Used:
	Account Name:		pekora
	Account Domain:		MIKO
	Logon GUID:		{00000000-0000-0000-0000-000000000000}

Target Server:
	Target Server Name:	localhost
	Additional Information:	localhost

Process Information:
	Process ID:		0x224
	Process Name:		C:\Windows\System32\lsass.exe

Network Information:
	Network Address:	10.120.116.21
	Port:			40454

4634:

An account was logged off.

Subject:
	Security ID:		MIKO\pekora
	Account Name:		pekora
	Account Domain:		MIKO
	Logon ID:		0x236AFD

Logon Type:			3

4776:

The computer attempted to validate the credentials for an account.

Authentication Package:	MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account:	pekora
Source Workstation:	WIN-9ITB345Q7UU
Error Code:	0x0

Anonymous Access:

Anonymous user access is only logged in Jenkins whenever they navigate to a page without permissions.

Because the URL is tracked, these log entries can be used to track:

• Failed script execution attempts.

• Exploitation attempts.

• Recon activity from threat actor.

Exposed Jenkins:

If Jenkins is exposed to the internet with anonymous read access, anyone can view the Jenkins console output of executed jobs and harvest credentials that may be exposed in command lines.

This is a very common technique and many attackers will just download the console output of every project in an exposed Jenkins instance to scan for credentials offline.

Readable by anonymous users

Evidence of this may be found in the Jenkins docker logs.

Anonymous users attempting to access a resource that they don't have permission to view is logged by default in Jenkins and may be used to find console output scraping. Look for patterns of mass scanning by anonymous users to prove this.

Documentation of this technique and various others can be found in the links below.

GitHub - gquere/pwn_jenkins: Notes about attacking Jenkins serversGitHub

Jenkins SecurityHackTricks Cloud

Credential Exposure Check:

To ensure that there is no additional exposure of credentials in the log files, trufflehog should be run against the Jenkins_home directory or the Jenkins_home/jobs directory.

If there are any exposed credentials found, they should be rotated immediately and the logs for the application they provide access to should be investigated to ensure there is no unauthorized access.

#Offline Jenkins directory:
trufflehog filesystem /Jenkins_home/jobs > Output.txt

#Live Jenkins instance:
trufflehog jenkins --url https://jenkins.example.com --username admin --password admin

GitHub - trufflesecurity/trufflehog: Find credentials all over the placeGitHub

Logging:

• Default Logs:

  • Active Directory Plugin:

    • Failed authentications to valid Active Directory users

  • Anonymous users:

    • Whenever an anonymous user browses to a location they don't have access to, the attempt and URL is logged by default.

• Not Default:

  • Script Console output and commands not logged by default.

  • Successful user authentications and failed user authentications are not logged by default.

  • Job creation, changes, and executions are not logged in Jenkins Docker logs.

Log Recorders:

Access logs are NOT enabled by default, you need to enable it by creating a log recorder that is monitoring the following logger:

• jenkins.security.SecurityListener

The log recorder file will contain what logger target is being monitored and is located in the following path:

• /var/jenkins_home/logs/<log name>.xml

SecurityListener (Jenkins core 2.466 API)

Command Misconfigurations:

1. Jenkins is accessible from the internet without restrictions.

2. Credentials are exposed in workspaces or console output.

3. MFA is not enabled.

4. Weak username/passwords for users allowed to access Jenkins.

5. Jenkins is not patched and can be exploited.

6. Users are overprivileged and allowed to execute commands from the Script Console.

Recommendations:

• Least Privilege:

  • Jenkins service account should not have root access. Practice least privilege to avoid Groovy scripts from reading credentials.xml file.

  • Practice least privilege in Jenkins accounts, avoid "Logged in users can do anything" setting.

  • Any user with job creation and configuration privileges has shell access to the nodes because job configuration allows an arbitrary command to run on the nodes.

  • Do not allow anonymous access.

• Logging:

  • By default, Jenkins does not enable access logging, this must be turned on.

  • Use logging plugins like "Audit Log" and "Audit Trail" to track job execution, script console execution, or login events.

• CVEs:

  • Ensure Jenkins and its plugins are updated to avoid exploitation of CVEs.

• Access Control:

  • Ensure Jenkins is not exposed to the internet and if it is, only allowlist specific IPs that should have access to it.

Managing SecurityManaging Security

Securing JenkinsSecuring Jenkins