Designing System Configurations in Puppet

👉Usage of Puppet Code

Modules are used to store puppet code. Use the tested and pre-built modules on Puppet Forge, a repository of hundreds of modules created by Puppet developers and the Puppet community.

Modules control particular infrastructure operations like installing and configuring applications. Code and data are both found in modules. What enables you to alter your configuration is the data.

You can isolate the data from the code and store it in one place using a tool named Hiera. This enables you to establish known parameters and variants, guardrails, and parameters so that your code is thoroughly tested and you can evaluate all the parameter edge cases.

👉Designing System Configurations in Puppet

Designing System Configurations in Puppet typically involves managing all of the software, services, and configuration necessary for a user on a specific system.

➡️The Roles and Profiles Method

When roles and profiles are absent, individuals often create system settings in their main manifest or node classifier and use Hiera to solve complex inheritance issues. Creating a group of related nodes and giving them classes is a common strategy. Subsequently, child groups with more classes are created for nodes with special requirements. Another typical strategy is to use a wide hierarchy that reflects all variations in the infrastructure and places everything in Hiera.

Perhaps you won't require roles and profiles if this works for you. However, most people find that the direct building becomes more complex to comprehend and manage with time.

The roles and profiles technique can make the code reusable, reconfigurable, and refactorable. It also helps to keep complexity under control.

Configuring Roles and Profiles

The roles and profiles are two other indirection levels between the node classifier and the component modules. The code is divided into three tiers using the roles and profiles method:

Component modules: Regular modules that control a single piece of technology, like puppetlabs/apache.

Profiles: Wrapper classes configuring a layered technology stack using several component modules.

Roles: These wrapper classes combine different profiles to create a whole system configuration.

Although it may appear that these additional indirection layers increase complexity, they allow you the opportunity to create useful, business-specific interfaces to the configuration that is most important to you. A better user interface makes it simpler to use hierarchical data, read system configurations, and rework code.

From top to bottom, the node classifier gives one role class to a set of nodes. No more classes are required because the role governs the setup of the entire system. The role is not in any way modified by the node classifier.

The only thing the role class does is define some profile classes with include.

For example:

class role::jenkins::master {
    include profile::base
    include profile::server
    include profile::jenkins::master
 }

Each profile uses different component modules and built-in resource types to construct a layered technological stack.

Profiles can use Hiera, Puppet lookup, or the console to get configuration information.

Classes from component modules are never assigned directly to a node; instead, they are always specified via a profile. If a component class includes parameters, you must define them in the profile; you should never override component class parameters using Hiera or Puppet lookup.

Rules For Profile Classes

Don't use resource-like declarations on profiles unless you can safely include them more than once.
Profiles can include other profiles.
All of the class parameters for profiles' component classes are theirs. The component class shouldn't use a value from Hiera data; if the profile omits one, you must desire the default value. Refactor the profile if you need to set a class parameter that was previously left out.
A profile can obtain the data required to configure component classes in one of three ways:

⭕Hardcode a parameter if your company always uses the same value for it.

⭕If you are unable to hardcode it, try computing it using the data you already have.

⭕Finally, if you are unable to compute it, check your data. Find situations when more than one parameter can be retrieved from the response to a single query to reduce the need for lookups.

Rules For Role Classes

Roles should only use include to declare profile classes. Don't declare any common resources or component classes in a role. The choice of which profiles to use can optionally be made by roles using conditional logic.
Roles shouldn't have their class parameters.
Roles shouldn't assign any profiles' class parameters. (Data lookup handles all of those).
The name of a role should be based on the name used by your company for the kind of node it handles. As a result, it makes it logical to create a role called role::jenkins::master if you frequently refer to a machine as a "Jenkins master." However, if you refer to it as a web server, you should use a name other than role::nginx, such as role::web.

Methods For Data Lookup

Data lookup is necessary for profiles to obtain the configuration that they typically need. The automatic class parameter lookup is used by this profile to make data requests.

# Example Hiera data
profile::jenkins::jenkins_port: 8000
profile::jenkins::java_dist: jre
profile::jenkins::java_version: '8'
 
# Example manifest
class profile::jenkins (
  Integer $jenkins_port,
  String  $java_dist,
  String  $java_version
) {
# ...

This profile omits the parameters and uses the lookup function:

class profile::jenkins {
  $jenkins_port = lookup('profile::jenkins::jenkins_port', {value_type => String, default_value => '9091'})
  $java_dist    = lookup('profile::jenkins::java_dist',    {value_type => String, default_value => 'jdk'})
  $java_version = lookup('profile::jenkins::java_version', {value_type => String, default_value => 'latest'})
  # ...

➡️Roles and Profiles Example

On the JVM, Jenkins is a continuous integration (CI) program. A web front end is provided by the Jenkins master server, which also executes CI tasks on demand or in response to events. In this example, we control how the Jenkins master servers are set up.

Prepare Your Environment

Prepare your environment before creating any new code, perform some additional setup if you're new to roles and profiles.

Create two modules: a role module and a profile module.

Put these two modules in your control repository rather than declaring them in your Puppetfile if you deploy your code using code manager or r10k because these two reserve the modules directory for their use.

Create a new directory called site in the repository.
To add the site to the modulepath, edit the environment.conf file. (For instance, use the site:modules:$basemodulepath as the modulepath).
Place the site directory's role and profile modules there.

2. Ensure that a hierarchy that works well for you is set up and functional using Hiera or Puppet search.

Selecting The Component Modules

In this example, we wish to use the puppet/jenkins module to manage Jenkins.

Java is necessary for Jenkins, and the puppet/jenkins module can take care of that automatically. But we'll disable that since we want more precise control over Java. So, puppetlabs/java is a viable option if we need a Java module.

That's sufficient to get started. As soon as those are operational, we can refactor and grow.

Write A Profile

According to Puppet, a profile is a regular class kept in the profile module. Create a brand-new class called profile::jenkins::master and place Puppet code in it at .../profile/manifests/jenkins/master.pp.

# /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp
class profile::jenkins::master (
  String $jenkins_port = '9091',
  String $java_dist    = 'jdk',
  String $java_version = 'latest',
) {


  class { 'jenkins':
    configure_firewall => true,
    install_java       => false,
    port               => $jenkins_port,
    config_hash        => {
      'HTTP_PORT'    => { 'value' => $jenkins_port },
      'JENKINS_PORT' => { 'value' => $jenkins_port },
    },
  }


  class { 'java':
    distribution => $java_dist,
    version      => $java_version,
    before       => Class['jenkins'],
  }
}

Set The Profile's Data

Suppose the following:

We employ a few unique facts:

➡ group: the group to which this node belongs. (This is typically a division inside our company or a significant undertaking shared by numerous nodes.)

➡ stage: This node's deployment stage (dev, test, or prod).

A five-layer hierarchy exists:

➡console_data for data defined in the console

➡nodes/%trusted.certname for per-node overrides

➡Group-specific data can be set using groups/%facts.group.

➡common for global fallback data.

Use Hiera or the configuration information in the console to enter the proper values in the data.

# /etc/puppetlabs/code/environments/production/data/nodes/ci-master01.example.com.yaml
 # --Nothing. We don't need any per-node values right now.


 # /etc/puppetlabs/code/environments/production/data/groups/ci/prod.yaml
 profile::jenkins::master::jenkins_port: '80'


 # /etc/puppetlabs/code/environments/production/data/groups/ci.yaml
 profile::jenkins::master::java_dist: 'oracle-jdk8'
 profile::jenkins::master::java_version: '8u92'


 # /etc/puppetlabs/code/environments/production/data/common.yaml
 # --Nothing. Just use the default parameter values.

Write A Role

profile::base- Every computer, including workstations, must have the role profile::base assigned to it. It oversees fundamental policies and employs some conditional reasoning as necessary to add OS-specific profiles.
profile::server- Every computer that offers a service across the network needs to be given to profile::server. It ensures operators can access the system and sets up timekeeping, firewalls, logging, and monitoring.

Therefore, those classes should also be included to manage one of our Jenkins masters.

class role::jenkins::master {
  include profile::base
  include profile::server
  include profile::jenkins::master
}

Assign The Role To Nodes

Finally, we give every node that serves as a Jenkins master the role::jenkins::master.

Use whichever tool you believe best serves your team to assign classes to nodes in Puppet. Your primary options are:

The console node classifier enables you to categorize groups of nodes based on their facts.
The primary manifest can assign classes using node statements or conditional logic.
Using Hiera or Puppet lookup, do a distinct array merging on a key associated with a specific class and then give the resulting array to the include method.

# /etc/puppetlabs/code/environments/production/manifests/site.pp
lookup('classes', {merge => unique}).include

➡️Designing Advanced Profiles

We gradually redesign or refactor our basic roles and profiles to address real-world issues. With a few slight modifications, the end outcome is the Jenkins profile that Puppet uses in production.

Here's the basic Jenkins profile we're starting with:

# /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp
class profile::jenkins::master (
  String $jenkins_port = '9091',
  String $java_dist    = 'jdk',
  String $java_version = 'latest',
) {


  class { 'jenkins':
    configure_firewall => true,
    install_java       => false,
    port               => $jenkins_port,
    config_hash        => {
      'HTTP_PORT'    => { 'value' => $jenkins_port },
      'JENKINS_PORT' => { 'value' => $jenkins_port },
    },
  }


  class { 'java':
    distribution => $java_dist,
    version      => $java_version,
    before       => Class['jenkins'],
  }
}

First Refactor: Split Out Java

Jenkins masters and agent nodes are what we wish to manage. We won't go into great depth on agent profiles, but the first problem is that they also require Java. We decided to create a distinct profile for Java. This way, we can manage it just once and then add the Java profile to the master and agent profiles.

Determine how customizable Java should be on Jenkins machines first. Puppet only uses two alternatives based on historical usage: install Oracle's Java 8 distribution or use OpenJDK 7, which the Jenkins module is in charge of. As a result, we can:

Make our new Java profile as straightforward as possible by hard coding Java 8 and requiring no configuration.

Replace the two Java parameters from profile::

jenkins::master with one Boolean parameter

Here's the new parameter list:

class profile::jenkins::master (
  String  $jenkins_port = '9091',
  Boolean $install_jenkins_java = true,
) { # …

This is how we decide which Java to use:

  class { 'jenkins':
    configure_firewall => true,
    install_java       => $install_jenkins_java,    # <--- here
    port               => $jenkins_port,
    config_hash        => {
      'HTTP_PORT'    => { 'value' => $jenkins_port },
      'JENKINS_PORT' => { 'value' => $jenkins_port },
    },
  }


  # Install Java 8 if you won't be using the Jenkins module's Java version. unless $install jenkins java {include profile::jenkins::usage::java8}

And our new Java profile:

::jenkins::usage::java8
# Sets up java8 for Jenkins on Debian
#
class profile::jenkins::usage::java8 {
  motd::register { 'Java usage profile (profile::jenkins::usage::java8)': }


  # OpenJDK 7 is already managed by the Jenkins module.
  # ::jenkins::install_java or ::jenkins::agent::install_java should be false to use this profile
  # the class parameter $install_jenkins_java can set this
  case $::osfamily {
    'debian': {
      class { 'java':
        distribution => 'oracle-jdk8',
        version      => '8u92',
      }


      package { 'tzdata-java':
        ensure => latest,
      }
    }
    default: {
      notify { "profile::jenkins::usage::java8 cannot set up JDK on ${::osfamily}": }

Diff Of First Refactor

@@ -1,13 +1,12 @@
 # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp
 class profile::jenkins::master (
-  String $jenkins_port = '9091',
-  String $java_dist    = 'jdk',
-  String $java_version = 'latest',
+  String  $jenkins_port = '9091',
+  Boolean $install_jenkins_java = true,
 ) {


   class { 'jenkins':
     configure_firewall => true,
-    install_java       => false,
+    install_java       => $install_jenkins_java,
     port               => $jenkins_port,
     config_hash        => {
       'HTTP_PORT'    => { 'value' => $jenkins_port },
@@ -15,9 +14,6 @@ class profile::jenkins::master (
     },
   }


-  class { 'java':
-    distribution => $java_dist,
-    version      => $java_version,
-    before       => Class['jenkins'],
-  }
+  # When not using the jenkins module's java version, install java8.
+  unless $install_jenkins_java  { include profile::jenkins::usage::java8 }
 }

Second Refactor: Manage The Heap

We at Puppet control the Jenkins app's Java memory size. Production servers lacked sufficient RAM for intensive use. Let's use the jenkins::sysconfig declared type from the Jenkins module to manage system properties:

# Control the heap size in MB on the master.
  if($::memorysize_mb =~ Number and $::memorysize_mb > 8192)
  {
    # anything over 8GB we should keep max 4GB for OS and others
    $heap = sprintf('%.0f', $::memorysize_mb - 4096)
  } else {
    # This is calculated as 50% of the total memory.
    $heap = sprintf('%.0f', $::memorysize_mb * 0.5)
  }
  # Set Java parameters, such as heap minimum and maximum sizes. See
  # https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties
  jenkins::sysconfig { 'JAVA_ARGS':
    value => "-Xms${heap}m -Xmx${heap}m -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"",
  }

Diff Of Second Refactor

@@ -16,4 +16,20 @@ class profile::jenkins::master (


   # When not using the jenkins module's java version, install java8.
   unless $install_jenkins_java  { include profile::jenkins::usage::java8 }
+
+  # Manage the heap size on the master, in MB.
+  if($::memorysize_mb =~ Number and $::memorysize_mb > 8192)
+  {
+    # anything over 8GB we should keep max 4GB for OS and others
+    $heap = sprintf('%.0f', $::memorysize_mb - 4096)
+  } else {
+    # This is calculated as 50% of the total memory.
+    $heap = sprintf('%.0f', $::memorysize_mb * 0.5)
+  }
+  # Set java params, like heap min and max sizes. See
+  # https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties
+  jenkins::sysconfig { 'JAVA_ARGS':
+    value => "-Xms${heap}m -Xmx${heap}m -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"",
+  }
+
 }

Third Refactor: Pin The Version

We begin by adding a parameter to manage upgrades. Now that we have changed a setting in.../data/groups/ci/dev.yaml without altering.../data/groups/ci.yaml, we can upgrade our development machines first and test that everything functions as expected before upgrading our production machines.

class profile::jenkins::master (
  Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb',
  # ...
) { # …
Then, we set the necessary parameters in the Jenkins class:
  class { 'jenkins':
    lts                => true,                    # <-- here
    repo               => true,                    # <-- here
    direct_download    => $direct_download,        # <-- here
    version            => 'latest',                # <-- here
    service_enable     => true,
    service_ensure     => running,
    configure_firewall => true,
    install_java       => $install_jenkins_java,
    port               => $jenkins_port,
    config_hash        => {
      'HTTP_PORT'    => { 'value' => $jenkins_port },
      'JENKINS_PORT' => { 'value' => $jenkins_port },
    },
  }

Diff Of Third Refactor

@@ -1,10 +1,17 @@
 # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp
 class profile::jenkins::master (
-  String  $jenkins_port = '9091',
-  Boolean $install_jenkins_java = true,
+  String                      $jenkins_port = '9091',
+  Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb',
+  Boolean                     $install_jenkins_java = true,
 ) {


   class { 'jenkins':
+    lts                => true,
+    repo               => true,
+    direct_download    => $direct_download,
+    version            => 'latest',
+    service_enable     => true,
+    service_ensure     => running,
     configure_firewall => true,
     install_java       => $install_jenkins_java,
     port               => $jenkins_port,

Fourth Refactor: Manually Manage The User Account

According to who needs to log into a particular computer, the profile::server class acquires virtual::users, which have a lot of virtual resources we can selectively realize.

Therefore, for this example, we modify the Jenkins profile to function similarly; we manage the jenkins user and the other user accounts. While we're doing that, we maintain a couple of directories that may be troublesome depending on how Jenkins is packed.

We're going to store some of the values in a params class, which is a class that sets shared variables but doesn't manage any resources because they are needed by Jenkins agents and masters alike.

  include profile::server
  # Some default values varying by OS:
  include profile::jenkins::params
  $jenkins_owner          = $profile::jenkins::params::jenkins_owner
  $jenkins_group          = $profile::jenkins::params::jenkins_group
  $master_config_dir      = $profile::jenkins::params::master_config_dir


  file { '/var/run/jenkins': ensure => 'directory' }


# Since the 'jenkins' user's homedir is managed by account::user class as 
#the `${master config dir}` directory, we must control `${master_config_dir/plugins}` here to #stop the upstream rtyler-jenkins module from attempting to handle the homedir as the #config dir. See the `manifests/plugin.pp` manifest for the upstream module.
  file { "${master_config_dir}/plugins":
    ensure  => directory,
    owner   => $jenkins_owner,
    group   => $jenkins_group,
    mode    => '0755',
    require => [Group[$jenkins_group], User[$jenkins_owner]],
  }


  Account::User <| tag == 'jenkins' |>


  class { 'jenkins':
    lts                => true,
    repo               => true,
    direct_download    => $direct_download,
    version            => 'latest',
    service_enable     => true,
    service_ensure     => running,
    configure_firewall => true,
    install_java       => $install_jenkins_java,
    manage_user        => false,                    # <-- here
    manage_group       => false,                    # <-- here
    manage_datadirs    => false,                    # <-- here
    port               => $jenkins_port,
    config_hash        => {
      'HTTP_PORT'    => { 'value' => $jenkins_port },
      'JENKINS_PORT' => { 'value' => $jenkins_port },
    },
  }

Diff Of Fourth Refactor

@@ -5,6 +5,33 @@ class profile::jenkins::master (
   Boolean                     $install_jenkins_java = true,
 ) {


+  # We rely on virtual resources that are ultimately declared by profile::server.
+  include profile::server
+
+  # Some default values that vary by OS:
+  include profile::jenkins::params
+  $jenkins_owner          = $profile::jenkins::params::jenkins_owner
+  $jenkins_group          = $profile::jenkins::params::jenkins_group
+  $master_config_dir      = $profile::jenkins::params::master_config_dir
+
+  file { '/var/run/jenkins': ensure => 'directory' }
+
+  # Because our account::user class manages the '${master_config_dir}' directory
+  # as the 'jenkins' user's homedir (as it should), we need to manage
+  # `${master_config_dir}/plugins` here to prevent the upstream
+  # rtyler-jenkins module from trying to manage the homedir as the config
+  # dir. For more info, see the upstream module's `manifests/plugin.pp`
+  # manifest.
+  file { "${master_config_dir}/plugins":
+    ensure  => directory,
+    owner   => $jenkins_owner,
+    group   => $jenkins_group,
+    mode    => '0755',
+    require => [Group[$jenkins_group], User[$jenkins_owner]],
+  }
+
+  Account::User <| tag == 'jenkins' |>
+
   class { 'jenkins':
     lts                => true,
     repo               => true,
@@ -14,6 +41,9 @@ class profile::jenkins::master (
     service_ensure     => running,
     configure_firewall => true,
     install_java       => $install_jenkins_java,
+    manage_user        => false,
+    manage_group       => false,
+    manage_datadirs    => false,
     port               => $jenkins_port,
     config_hash        => {
       'HTTP_PORT'    => { 'value' => $jenkins_port },

Fifth Refactor: Manage More Dependencies

To access private Git repositories and execute commands on Jenkins agent nodes, Jenkins always needs Git installed. It also needs SSH keys. Puppet also takes care of the Jenkins plugins from a standard list.

Managing Git is easy:

 package { 'git':
    ensure => present,
  }

SSH keys are more difficult because they contain sensitive information. We put them on a custom mount point on one particular Puppet server because we can't check them into version control with the rest of our Puppet scripts.

We established a rule for accessing it since this server differs from our standard Puppet servers: you cannot hardcode the hostname; instead, you must look it up from data. As a result, if the secure server ever moves, we can adjust only one location.

  $secure_server = lookup('puppetlabs::ssl::secure_server')


  file { "${master_config_dir}/.ssh":
    ensure => directory,
    owner  => $jenkins_owner,
    group  => $jenkins_group,
    mode   => '0700',
  }


  file { "${master_config_dir}/.ssh/id_rsa":
    ensure => file,
    owner  => $jenkins_owner,
    group  => $jenkins_group,
    mode   => '0600',
    source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins",
  }


  file { "${master_config_dir}/.ssh/id_rsa.pub":
    ensure => file,
    owner  => $jenkins_owner,
    group  => $jenkins_group,
    mode   => '0640',
    source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins.pub",
  }

Plugins are also challenging because we want to configure them manually on a few Jenkins masters. Therefore, we create a distinct profile for the base list and utilize a parameter to determine whether or not to use it.

class profile::jenkins::master (
  Boolean                     $manage_plugins = false,
  # ...
) {
  # ...
  if $manage_plugins {
    include profile::jenkins::master::plugins
  }

In the plugins profile, we can use the jenkins::plugin resource type provided by the Jenkins module.

# /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master/plugins.pp
class profile::jenkins::master::plugins {
  jenkins::plugin { 'audit2db':          }
  jenkins::plugin { 'credentials':       }
  jenkins::plugin { 'jquery':            }
  jenkins::plugin { 'job-import-plugin': }
  jenkins::plugin { 'ldap':              }
  jenkins::plugin { 'mailer':            }
  jenkins::plugin { 'metadata':          }
  # ... and so on.
}

Diff Of Fifth Refactor

@@ -1,6 +1,7 @@
 # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp
 class profile::jenkins::master (
   String                      $jenkins_port = '9091',
+  Boolean                     $manage_plugins = false,
   Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb',
   Boolean                     $install_jenkins_java = true,
 ) {
@@ -14,6 +15,20 @@ class profile::jenkins::master (
   $jenkins_group          = $profile::jenkins::params::jenkins_group
   $master_config_dir      = $profile::jenkins::params::master_config_dir


+  if $manage_plugins {
+    # About 40 jenkins::plugin resources:
+    include profile::jenkins::master::plugins
+  }
+
+  # Sensitive info (like SSH keys) isn't checked into version control like the
+  # rest of our modules; instead, it's served from a custom mount point on a
+  # designated server.
+  $secure_server = lookup('puppetlabs::ssl::secure_server')
+
+  package { 'git':
+    ensure => present,
+  }
+
   file { '/var/run/jenkins': ensure => 'directory' }


   # Because account::user class manages the '${master_config_dir}' directory
@@ -69,4 +84,29 @@ class profile::jenkins::master (
     value => "-Xms${heap}m -Xmx${heap}m -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"",
   }


+ #Install the SSH keys that Jenkins need to control its agent computers and access Git  
+ #repositories.
+  file { "${master_config_dir}/.ssh":
+    ensure => directory,
+    owner  => $jenkins_owner,
+    group  => $jenkins_group,
+    mode   => '0700',
+  }
+
+  file { "${master_config_dir}/.ssh/id_rsa":
+    ensure => file,
+    owner  => $jenkins_owner,
+    group  => $jenkins_group,
+    mode   => '0600',
+    source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins",
+  }
+
+  file { "${master_config_dir}/.ssh/id_rsa.pub":
+    ensure => file,
+    owner  => $jenkins_owner,
+    group  => $jenkins_group,
+    mode   => '0640',
+    source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins.pub",
+  }
+
 }

Sixth Refactor: Manage Logging And Backups

Typically, having a backup is a good idea. We can use the backup::job resource type provided by our in-house backup module (profile::server takes care of its prerequisites). But if someone is setting up a temporary Jenkins installation to test something, we should make backups optional to prevent them from unintentionally uploading trash to our backup server.

class profile::jenkins::master (
  Boolean                     $backups_enabled = false,
  # ...
) {
  # ...
  if $backups_enabled {
    backup::job { "jenkins-data-${::hostname}":
      files => $master_config_dir,
    }
  }
}

Use jenkins::sysconfig and our homegrown logrotate::job

class profile::jenkins::master (
  Optional[String[1]]         $jenkins_logs_to_syslog = undef,
  # ...
) {
  # ...
  if $jenkins_logs_to_syslog {
    jenkins::sysconfig { 'JENKINS_LOG':
      value => "$jenkins_logs_to_syslog",
    }
  }
  # ...
  logrotate::job { 'jenkins':
    log     => '/var/log/jenkins/jenkins.log',
    options => [
      'daily',
      'copytruncate',
      'missingok',
      'rotate 7',
      'compress',
      'delaycompress',
      'notifempty'
    ],
  }
}

Diff Of Sixth Refactor

@@ -1,8 +1,10 @@
 # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp
 class profile::jenkins::master (
   String                      $jenkins_port = '9091',
+  Boolean                     $backups_enabled = false,
   Boolean                     $manage_plugins = false,
   Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb',
+  Optional[String[1]]         $jenkins_logs_to_syslog = undef,
   Boolean                     $install_jenkins_java = true,
 ) {


@@ -84,6 +86,15 @@ class profile::jenkins::master (
     value => "-Xms${heap}m -Xmx${heap}m -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"",
   }




+ #Jenkins master logs should be sent to syslog. When a facility is selected, 
+ # the jenkins log uses that value rather than another log file, such as daemon.info.
+  if $jenkins_logs_to_syslog {
+    jenkins::sysconfig { 'JENKINS_LOG':
+      value => "$jenkins_logs_to_syslog",
+    }
+  }
+
   # Deploy the SSH keys that Jenkins needs to manage its agent machines and
   # access Git repos.
   file { "${master_config_dir}/.ssh":
@@ -109,4 +120,29 @@ class profile::jenkins::master (
     source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins.pub",
   }


+  # Back up Jenkins' data.
+  if $backups_enabled {
+    backup::job { "jenkins-data-${::hostname}":
+      files => $master_config_dir,
+    }
+  }
+
+  # (QENG-1829) Logrotate rules:
+ # Jenkins' default logrotate config keeps 52 weeks' worth of logs and rotates jenkins.log  
+ # just once each week. Let's rotate the logs every day and remove them after 7 days to 
+ # save disc use since we hardly ever look at the logs.
+  logrotate::job { 'jenkins':
+    log     => '/var/log/jenkins/jenkins.log',
+    options => [
+      'daily',
+      'copytruncate',
+      'missingok',
+      'rotate 7',
+      'compress',
+      'delaycompress',
+      'notifempty'
+    ],
+  }
+
 }

Seventh Refactor: Use A Reverse Proxy For HTTPS

An Nginx reverse proxy will enable us to use HTTPS for the Jenkins web interface. Additionally, we wish to standardize the ports: the proxy always forwards the traffic over port 443 for HTTPS and port 80 for HTTP, while the Jenkins app always binds to its default port.

We can provide a $ssl parameter if we wish to keep plain HTTP available. Jenkins can be accessed through HTTP and HTTPS if this setting is false (the default). To allow the proxy to listen on a hostname other than the node's primary FQDN, we can add the $site alias argument.

class profile::jenkins::master (
  Boolean                     $ssl = false,
  Optional[String[1]]         $site_alias = undef,
  # IMPORTANT: notice that $jenkins_port is removed.
  # ...

Set configure_firewall => false in the Jenkins class:

  class { 'jenkins':
    lts                => true,
    repo               => true,
    direct_download    => $direct_download,
    version            => 'latest',
    service_enable     => true,
    service_ensure     => running,
    configure_firewall => false,                # <-- here
    install_java       => $install_jenkins_java,
    manage_user        => false,
    manage_group       => false,
    manage_datadirs    => false,
    # IMPORTANT: notice that port and config_hash are removed.
  }

Where Nginx can access them, we must install SSL certificates. We already had a profile for that since we frequently deliver content via HTTPS:

  # Deploy the SSL certificate/chain/key for sites on this domain.
  include profile::ssl::delivery_wildcard

Add some info for the message of the day, handled by puppetlabs/motd

  motd::register { 'Jenkins CI master (profile::jenkins::master)': }


  if $site_alias {
    motd::register { 'jenkins-site-alias':
      content => @("END"),
                 profile::jenkins::master::proxy

                 Jenkins site alias: ${site_alias}
                 |-END
      order   => 25,
    }
  }

A new profile with the name profile::jenkins::master::proxy is responsible for most of the work. We've omitted the code for simplicity; in brief, it accomplishes the following:

Embrace the nginx profile.
Create a vhost using the resource types provided by jfryman/nginx, and if vanilla HTTP isn't enabled, force a redirect to HTTPS.
For access and error logs, configure logstash forwarding.
To manage firewall rules, if necessary, include profile::fw::https.

In our primary profile, we then declare that profile.

  class { 'profile::jenkins::master::proxy':
    site_alias  => $site_alias,
    require_ssl => $ssl,
  }

Diff Of Seventh Refactor

@@ -1,8 +1,9 @@
 # /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp
 class profile::jenkins::master (
-  String                      $jenkins_port = '9091',
   Boolean                     $backups_enabled = false,
   Boolean                     $manage_plugins = false,
+  Boolean                     $ssl = false,
+  Optional[String[1]]         $site_alias = undef,
   Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb',
   Optional[String[1]]         $jenkins_logs_to_syslog = undef,
   Boolean                     $install_jenkins_java = true,
@@ -11,6 +12,9 @@ class profile::jenkins::master (
   # We rely on virtual resources that are ultimately declared by profile::server.
   include profile::server


+  # Deploy the SSL certificate/chain/key for sites on this domain.
+  # include profile::ssl::delivery_wildcard
+
   # Some default values that vary by OS:
   include profile::jenkins::params
   $jenkins_owner          = $profile::jenkins::params::jenkins_owner
@@ -22,6 +26,31 @@ class profile::jenkins::master (
     include profile::jenkins::master::plugins
   }


+  motd::register { 'Jenkins CI master (profile::jenkins::master)': }
+
+  # This adds the site_alias to the message of the day for convenience when
+  # logging into a server via FQDN. Because of the way motd::register works, we
+  # need a sort of funny formatting to put it at the end (order => 25) and to
+  # list a class so there isn't a random "--" at the end of the message.
+  if $site_alias {
+    motd::register { 'jenkins-site-alias':
+      content => @("END"),
+                 profile::jenkins::master::proxy
+
+                 Jenkins site alias: ${site_alias}
+                 |-END
+      order   => 25,
+    }
+  }
+
+  # This is a "private" profile that sets up an Nginx proxy -- it's only ever
+  # declared in this class, and it would work identically pasted inline.
+  # But because it's long, this class reads more cleanly with it separated out.
+  class { 'profile::jenkins::master::proxy':
+    site_alias  => $site_alias,
+    require_ssl => $ssl,
+  }
+
#Instead of being checked into version control like the rest of our modules, 
#sensitive information (such SSH keys) is served from a specific mount point on a chosen server.


@@ -56,16 +85,11 @@ class profile::jenkins::master (
     version            => 'latest',
     service_enable     => true,
     service_ensure     => running,
-    configure_firewall => true,
+    configure_firewall => false,
     install_java       => $install_jenkins_java,
     manage_user        => false,
     manage_group       => false,
     manage_datadirs    => false,
-    port               => $jenkins_port,
-    config_hash        => {
-      'HTTP_PORT'    => { 'value' => $jenkins_port },
-      'JENKINS_PORT' => { 'value' => $jenkins_port },
-    },
   }


# When not using the jenkins module's java version, install java8.

After all the refactoring, this is what the final code for profile::jenkins::master looks like:

# /etc/puppetlabs/code/environments/production/site/profile/manifests/jenkins/master.pp
# Class: profile::jenkins::master
#
# Install a Jenkins master that satisfies the requirements of Puppet.
#
class profile::jenkins::master (
  Boolean                     $backups_enabled = false,
  Boolean                     $manage_plugins = false,
  Boolean                     $ssl = false,
  Optional[String[1]]         $site_alias = undef,
  Variant[String[1], Boolean] $direct_download = 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.2_all.deb',
  Optional[String[1]]         $jenkins_logs_to_syslog = undef,
  Boolean                     $install_jenkins_java = true,
) {


  # We rely on virtual resources that are ultimately declared by profile::server.
  include profile::server


  # Deploy the SSL certificate/chain/key for sites on this domain.
  include profile::ssl::delivery_wildcard


  # Some default values that vary by OS:
  include profile::jenkins::params
  $jenkins_owner          = $profile::jenkins::params::jenkins_owner
  $jenkins_group          = $profile::jenkins::params::jenkins_group
  $master_config_dir      = $profile::jenkins::params::master_config_dir


  if $manage_plugins {
    # About 40 jenkins::plugin resources:
    include profile::jenkins::master::plugins
  }


  motd::register { 'Jenkins CI master (profile::jenkins::master)': }


# This makes it easier to log onto a server using a FQDN by adding the site_alias to the 
# message of the day.. Because of the way motd::register works, we
  # need a sort of funny formatting to put it at the end (order => 25) and to
  # list a class so there isn't a random "--" at the end of the message.
  if $site_alias {
    motd::register { 'jenkins-site-alias':
      content => @("END"),
                 profile::jenkins::master::proxy


                 Jenkins site alias: ${site_alias}
                 |-END
      order   => 25,
    }
  }


  # This "private" profile creates a Nginx proxy; it is only ever declared in this class, and if 
# pasted inline, it would function exactly the same. However, because it is lengthy, it reads 
# better when it is divided apart..
  class { 'profile::jenkins::master::proxy':
    site_alias  => $site_alias,
    require_ssl => $ssl,
  }


# Instead of being checked into version control like the rest of our modules, sensitive 
# information (such SSH keys) is served from a specific mount point on a chosen server.
  $secure_server = lookup('puppetlabs::ssl::secure_server')


  # Dependencies:
  #   - Pull in apt if we're on Debian.
  #   - Pull in the 'git' package, used by Jenkins for Git polling.
  #   - Manage the 'run' directory (fix for busted Jenkins packaging).
  if $::osfamily == 'Debian' { include apt }


  package { 'git':
    ensure => present,
  }


  file { '/var/run/jenkins': ensure => 'directory' }


  # Because our account::user class manages the '${master_config_dir}' directory
  # as the 'jenkins' user's homedir (as it should), we need to manage
  # `${master_config_dir}/plugins` here to prevent the upstream
  # rtyler-jenkins module from trying to manage the homedir as the config
  # dir. For more info, see the upstream module's `manifests/plugin.pp`
  # manifest.
  file { "${master_config_dir}/plugins":
    ensure  => directory,
    owner   => $jenkins_owner,
    group   => $jenkins_group,
    mode    => '0755',
    require => [Group[$jenkins_group], User[$jenkins_owner]],
  }


  Account::User <| tag == 'jenkins' |>


  class { 'jenkins':
    lts                => true,
    repo               => true,
    direct_download    => $direct_download,
    version            => 'latest',
    service_enable     => true,
    service_ensure     => running,
    configure_firewall => false,
    install_java       => $install_jenkins_java,
    manage_user        => false,
    manage_group       => false,
    manage_datadirs    => false,
  }


  # When not using the jenkins module's java version, install java8.
  unless $install_jenkins_java  { include profile::jenkins::usage::java8 }


  # Manage the heap size on the master, in MB.
  if($::memorysize_mb =~ Number and $::memorysize_mb > 8192)
  {
    # anything over 8GB we should keep max 4GB for OS and others
    $heap = sprintf('%.0f', $::memorysize_mb - 4096)
  } else {
    # This is calculated as 50% of the total memory.
    $heap = sprintf('%.0f', $::memorysize_mb * 0.5)
  }
  # Set java parameters, like heap minimum and maximum sizes. See
  # https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties
  jenkins::sysconfig { 'JAVA_ARGS':
    value => "-Xms${heap}m -Xmx${heap}m -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Dhudson.model.DirectoryBrowserSupport.CSP=\\\"default-src 'self'; img-src 'self'; style-src 'self';\\\"",
  }


  # Forward jenkins master logs to syslog.
  # When set to facility.level the jenkins_log uses that value instead of a
  # separate log file, for example daemon.info
  if $jenkins_logs_to_syslog {
    jenkins::sysconfig { 'JENKINS_LOG':
      value => "$jenkins_logs_to_syslog",
    }
  }


# Install the SSH keys that Jenkins need to control its agent systems and access Git 
# repositories.
  file { "${master_config_dir}/.ssh":
    ensure => directory,
    owner  => $jenkins_owner,
    group  => $jenkins_group,
    mode   => '0700',
  }


  file { "${master_config_dir}/.ssh/id_rsa":
    ensure => file,
    owner  => $jenkins_owner,
    group  => $jenkins_group,
    mode   => '0600',
    source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins",
  }


  file { "${master_config_dir}/.ssh/id_rsa.pub":
    ensure => file,
    owner  => $jenkins_owner,
    group  => $jenkins_group,
    mode   => '0640',
    source => "puppet://${secure_server}/secure/delivery/id_rsa-jenkins.pub",
  }


  # Back up Jenkins' data.
  if $backups_enabled {
    backup::job { "jenkins-data-${::hostname}":
      files => $master_config_dir,
    }
  }


  # (QENG-1829) Logrotate rules:
# Jenkins' default logrotate configuration keeps 52 weeks' worth of logs and rotates 
# jenkins.log just once each week. Let's rotate the logs every day and remove them after 7 
# days to save disc use since we hardly ever look at the logs.
  logrotate::job { 'jenkins':
    log     => '/var/log/jenkins/jenkins.log',
    options => [
      'daily',
      'copytruncate',
      'missingok',
      'rotate 7',
      'compress',
      'delaycompress',
      'notifempty'
    ],
  }
}

➡️Designing Convenient Roles

There are various methods for creating roles, and you must choose the one that works best for you and your team.

Start with specific roles and only move from them in small, careful stages.

Here is the fundamental Jenkins role to start with:

class role::jenkins::master {
  include profile::base
  include profile::server
  include profile::jenkins::master
}

First Approach: Granular Roles

Making one role for each type of node is the simplest strategy. On their Jenkins masters, for instance, the Puppet Release Engineering (RE) team maintains a few more resources.

We would have at least two Jenkins master roles if we used granular roles.

class role::jenkins::master {
  include profile::base
  include profile::server
  include profile::jenkins::master
}

The advantages of this configuration are:

Readability: You can quickly identify which profiles go into each sort of node by glancing at a single class.
Each position is simply a collection of profiles organized linearly.

Some negatives include:

Role bloat occurs when many nodes are only marginally distinct from one another.
Repetition: With one exception, the two jobs described above are nearly identical. It's more challenging to understand their relationship if they are two distinct positions, and changing them might be more irksome.

Second Approach: Conditional Logic

As an alternative, conditional logic can be used to address distinctions between types of nodes that are closely related.

class role::jenkins::master::release {
  include profile::base
  include profile::server
  include profile::jenkins::master

 if $facts['group'] == 'release' {
    include profile::jenkins::master::release
  }
}

The advantages of this strategy are:

You have fewer roles in managing, and they are simple.

The disadvantages are:

Even in this straightforward situation, conditional logic is usually easy to grasp but accounts for complex roles; you could be tempted to add a tonne of new unique facts. Because a reader also needs to understand what those facts signify, making roles much more difficult to read.

Third approach: Nested Roles

Allowing roles to overlap with other roles is another strategy for decreasing duplication.

class role::jenkins::master {
  # Parent role:
  include role::server
  # Unique classes:
  include profile::jenkins::master
}


class role::jenkins::master::release {
  # Parent role:
  include role::jenkins::master
  # Unique classes:
  include profile::jenkins::master::release
}

The advantages of this strategy are:

You have fewer roles in managing, and they are simple.
Heightened awareness in your node classifier.

The disadvantages are:

Decreased readability: More files need to be opened to see a role's actual content. If you only go one level deep, this isn't too much of an issue, but reaching three or four levels can become difficult.

Fourth Approach: Multiple Roles Per Node

We advise you to provide each node with a single role in general. That's the optimal method to operate in an architecture where nodes typically offer a single primary service.

However, it may make sense to give numerous roles if your nodes frequently offer more than one primary function.

Consider the case of an extensive application that typically consists of a web server, a database server, and an application server.

You've decided to give your developers access to an "all-in-one" node type to facilitate simpler testing while developing. You could accomplish this by creating a new class called role::our_application::monolithic that contains all of the profiles that make up the three standard roles, but you might find it easier to use your node classifier to give those all-in-one machines the roles role::our application::app, role::our_application::db, and role::our_application::web.

Advantages:

This strategy has the advantage of having fewer jobs and being simple to manage.

Disadvantages:

The disadvantages are that there isn't a single "role" that characterizes your multi-purpose nodes; rather, the information about what's on them is dispersed throughout your roles and your node classifier, and you have to cross-reference to comprehend their settings. This makes text harder to read.
Your "normal" roles may need to be more complex because there may be other minute variations between the normal and all-in-one versions of a complex application. Even though it increases the number of roles and introduces repetition, creating a distinct role for this type of node would likely lessen your overall complexity.

Fifth Approach: Super Profiles

Since profiles can already contain other profiles, you may elect to impose a further requirement at your company: all profiles must have any other profiles required to administer a full node that offers that service.

You may manage a Jenkins master server by explicitly assigning profile::jenkins::master in your node classifier. For example, Puppet's profile::jenkins::master class might comprise both profile::server and profile::base. In other words, the roles layer would no longer be required because the "primary" profile would perform all of the tasks that a role typically performs.

The advantages of this strategy are:

This makes the dependency chain for a complex service more obvious.
This may be made a lot simpler in several ways, depending on how you think of coding!

The disadvantages are:

Decrease in adaptability. As a result, fewer combinations of your roles are possible, and you are less able to use alternative implementations of dependencies for nodes with various requirements.
Decreased readability. Similar to nested roles, you lose the benefit of having a clear, concise list of the components that make up a node. You also lose the distinct distinction between "top-level" complete system configurations (roles) and "mid-level" collections of technologies, in contrast to nested roles (profiles).

Sixth Approach: Building Roles In The Node Classifier

You might discover that your node classifier is adaptable enough to develop roles directly rather than using the Puppet language to create them and then assign them to nodes.

To accomplish the same task as our fundamental role::jenkins::master class, you could, for instance, create a "Jenkins masters" group in the console and assign it the profile::base, profile::server, and profile::jenkins::master classes.

The advantages of this strategy are:

Your node classifier gains strength and can serve as the coordination hub for controlling nodes.
Better readability: Without the need to cross-reference with manifests in your role module, a node's terminal page displays the full content of its role.

The disadvantages are:

Decrease in flexibility: Conditional logic in the Puppet language is frequently more adaptable and practical than node classifiers in general, including the console.
It's more challenging to ensure your roles follow the same code promotion procedures and are no longer stored in the same code repository as your profiles.

👉About Hiera in Puppet

Puppet employs Hiera for two purposes:

Key-value pairs should be used to store the configuration data.
Look up the information a specific module requires for a particular node while compiling the catalog.

This is achieved by:

Auto Parameter searching for classes in the catalog
Calls for explicit lookups

The "defaults, with overrides" pattern is used by Hiera's hierarchical lookups, allowing you to declare common data just once and override it when the default doesn't function. Hiera specifies data sources using Puppet facts, allowing you to organize overrides to match your infrastructure. Although employing facts for this purpose is typical, data sources can also be defined without utilizing facts.

👉Puppet Language

To specify the intended state of your system, the user can use the declarative language provided by Puppet.

In documents referred to as manifests, the user will outline the ideal condition of the system. Manifests specify how the operating system and network resources should be set up, including files, packages, and services. Puppet then organizes these manifests into catalogs and applies each catalog to the associated node in the user's infrastructure to ensure the node is configured correctly.

The evaluation order is crucial to several aspects of the Puppet language. Variables, for instance, need to be set before they may be referenced. Puppet highlights places in the language reference where the sequencing of sentences is essential.

We hope you have learned everything about the Designing System Configurations in Puppet.

You can also check about Java Tokens here.

Frequently Asked Questions

What is Puppet?

With the aid of Puppet, you can manage and automate the configuration of servers. When using Puppet, specify the ideal state for the infrastructure systems you want to manage.

What is DevOps?

DevOps is a collection of cultural ideas, operational procedures, and technical resources that rapidly enhance an organization's ability to deliver products and services.

What language is Puppet built on?

Puppet is built using Ruby’s domain-specific language.

Is Puppet an open-source tool?

Puppet is an open-source tool. Puppet is also available in the enterprise version.

What are the requirements for designing system configurations in Puppet?

For Designing System Configurations in Puppet, the roles and profile method, designing advanced profiles and designing convenient roles are needed.

Conclusion

In this blog, we explored the Designing System Configurations in Puppet and learned about the Life Cycle of Puppet run. We further discussed Puppet Code usage and Hiera in Puppet. We then discussed the Puppet language.

We hope this blog has helped you enhance your knowledge of Designing System Configurations in Puppet and if you want to learn more, check out our articles here.

You can refer to other similar articles as well

Refer to our guided paths on Coding Ninjas Studio to learn more about DSA, Competitive Programming, JavaScript, System Design, etc. Enroll in our courses and refer to the mock test and problems available. Also, look at the interview experiences for placement preparations.

Happy Learning Ninja! 🥷

Designing System Configurations in Puppet

Introduction⛳

👉Life Cycle of Puppet Run

Agent-Server HTTPS communications

Catalog Compilation

Static Catalogs

👉Usage of Puppet Code

👉Designing System Configurations in Puppet

➡️The Roles and Profiles Method

Configuring Roles and Profiles

Rules For Profile Classes

Rules For Role Classes

Methods For Data Lookup

➡️Roles and Profiles Example

Prepare Your Environment

Selecting The Component Modules

Write A Profile

Set The Profile's Data

Write A Role

Assign The Role To Nodes

➡️Designing Advanced Profiles

First Refactor: Split Out Java

Second Refactor: Manage The Heap

Third Refactor: Pin The Version

Fourth Refactor: Manually Manage The User Account

Fifth Refactor: Manage More Dependencies

Sixth Refactor: Manage Logging And Backups

Seventh Refactor: Use A Reverse Proxy For HTTPS

➡️Designing Convenient Roles

First Approach: Granular Roles

Second Approach: Conditional Logic

Third approach: Nested Roles

Fourth Approach: Multiple Roles Per Node

Fifth Approach: Super Profiles

Sixth Approach: Building Roles In The Node Classifier

👉About Hiera in Puppet

👉Puppet Language

Frequently Asked Questions

What is Puppet?

What is DevOps?

What language is Puppet built on?

Is Puppet an open-source tool?

What are the requirements for designing system configurations in Puppet?

Conclusion