Tag Archives: RHCE

Back to Black

I am back to using Puppet with my home lab, after just about a year running SaltStack.

I enjoyed using Saltstack, with its light-weight orchestration and fairly simple syntax. However, I am planning to get my RHCA by the end of next year, and one of the exams I am planning to take (EX405) to satisfy the requirement is based on open-source Puppet.

Essentially, in order to pass the exam, I must be able to:

  • Install and configure Puppet.
  • Create and maintain Puppet manifests.
  • Create Puppet modules.
  • Use facter to obtain system information.
  • Work with Git repositories.
  • Implement Puppet in a Red Hat Satellite 6 environment.

With the exception of the last one, I am pretty comfortable with the rest of the objectives. However, I won’t make the same mistake I did with the RHCE exam by taking it for granted; so for the 12-18 month, I will be heavily managing my home lab with Puppet.

On that note, even though it had been over a year since I last touched a Puppet install, it was surprisingly easy to get back up to speed (yay for muscle memory). I was able to quickly get Puppet Server installed, configure R10k and deployed PuppetDB.

I did not put  MCollective on, though – and I am not planning to anytime soon. After months of using with Saltstack, I found MCollective to be quite limited for the amount of resources it consumes. Instead, for orchestration, I’ll leverage Ansible to kick off my puppet runs, which, incidentally, is another expertise exam that is coming from Red Hat.

At this point, now my environment has been setup, it is time to start planning some Puppet-based projects. Stay tuned!

It’s bold strategy, Cotton. Let see if it pays off

A month ago, I took the RHCE exam, fairly confident I will pass after spending hours studying and practicing.

It did not happen. I was not even able to complete the exam on time. I was hoping when I came home that I somehow squeaked though, but then came the exam notification::

Passing score for the exam: 210
Your score: 206

Result: NO PASS

For the next few hours, I was pretty depressed. I actually studied for the test far more extensively than the last time I took the RHCE, so it was a big blow to my confidence. At one point, I thought about not continuing on the RHCA path.

Then I decided to re-group and give it another go.

After signing up for the exam again (which, I will add, came at considerable cost, as Red Hat do not offer free re-takes), I took another look at the exam objectives and realize that in order to pass the exam, I need to complete all objectives in 3 1/2 hours (or 210 minutes). So I consolidated the list of objects as follows:

  • Configure a caching-only name server
  • Configure a system to forward all email to a central mail server
  • SSH Key Configuration with ACL
  • Synchronize time using other NTP peers
  • Apache – Configure a virtual host – with acl
  • Apache – Configure private directories
  • Apache – Configure group-managed content
  • Apache – Deploy a basic CGI application
  • Apache – Configure TLS security
  • Produce and deliver reports on system utilization (processor, memory, disk, and network)
  • Configure a system to authenticate using Kerberos
  • NFS – Provide network shares to specific clients
  • NFS – Provide network shares suitable for group collaboration (multi-user)
  • NFS – Use Kerberos to control access to NFS network shares
  • Samba – Provide network shares to specific clients
  • Samba – Provide network shares suitable for group collaboration
  • Use firewalld and associated mechanisms such as rich rules, zones and custom rules, to implement packet filtering and configure network address translation (NAT)
  • Route IP traffic and create static routes
  • Use /proc/sys and sysctl to modify and set kernel runtime parameters
  • Configure IPv6 addresses and perform basic IPv6 troubleshooting
  • Use network teaming or bonding to configure aggregated network links between two Red Hat Enterprise Linux systems
  • Install and configure MariaDB
  • Use shell scripting to automate system maintainance tasks
  • Configure a system as either an iSCSI target or initiator that persistently mounts an iSCSI target

Then I put them spreadsheet and start logging the time it takes me to complete each task over the course the week. The results were not pretty – it took about 162 minutes complete most of them.

(Actually, some of the tasks (in particularly, Apache), took far longer than I expected and some others I gave up after 10-15 minutes).

The important thing, though, is after that practice run, I know where my area sof weaknesses were. So I review the material again on my way to work and back, did some quick practice sessions and then went through the tasks again.

As the result, the following week was a different story I was able to cut my time down by almost 40 minutes – down to 128 minutes.

Again, I look at areas where I was weak at, practice and review. By the Sunday before the exam, I was able to cut my time to under 2 hours. Then I did some final review on some parts on Sunday and Monday.

As the result, when I re-took the exam Tuesday afternoon, I was able to breeze through all the items, and complete all of them with an hour to spare. At that point, I was able to spend the remaining time validating the setup, and going back and correcting things that I missed.

Later on that evening, I received the results:


Passing score for the exam: 210
Your score: 271

Result: PASS

Boom, baby.

I should put something here

Some days, keeping up with technology can be a mix of frustration and excitement.

I am currently working on getting back my RHCE (Red Hat Certified Engineer) credentials (I had it before, but for reasons I won’t get to, it expired). From there, I will be able to avail myself of a suite of certificates from Red Hat, eventually getting either a RHCA (Red Hat Certified Architect) in Cloud or DevOps (or if time does not permits, just plain RHCA) I will do this by using existing resources (books, documentation and supplemented by in-expensive online training) rather that taking the rather pricey ROLE courses.

That is the idea, at least.

Case in point, Samba. Now, I don’t use Samba that much, but it is a key objective to complete in the RHCE exam – not just using it, but configuring and setting up the appropriate access controls. From reading the RHCE books, it seems pretty straight forward. For example:

  • Provide network shares to specific clients
  • Provide network shares suitable for group collaboration

Which mean you need to do the following on the server:

1) Install Samba on the server.

yum -y install samba samba-client

2) Add group that will be used for collaboration

groupadd -g 8888 shared

3) Modify existing users so they are part of the group

usermod -aG shared amy
usermod -aG shared rory

5) Create samba users:

smbpasswd -a amy
smbpasswd -a rory

6) Set the appropriate permissions on the directory you want to share.

chmod 770 /srv/directory_to_be_shared
chown nobody:shared /srv/directory_to_be_shared

7) Set selinux permissions as follows:

semanage fcontext -a -t samba_public_t /srv/directory_to_be_shared
restorecon -rv /srv/directory_to_be_shared

8) Create entry in /etc/samba/smb.conf

comment = “shared directory”
path = /srv/directory_to_be_shared
writable = yes
browsable = yes
write list = +shared
hosts allow = foo.bar.monzell.com

9) run testparm to validate the configuration

10) Enable and start samba:

systemctl enable samba
systemctl start samba

11) open the firewall:

firewall-cmd —add-service=samba
firewall-cmd —add-service=samba —permanent

While on the client:

1) Install samba and cifs-utils:

yum -y install cifs-utils samba

2) Create directory to mount the share:

mkdir /mnt/shared

3) Create a file that contain the credentials used to mount the share and secure the file:

echo 'username=amy' > /etc/samba/secret
echo 'password=doctor!' >> /etc/samba/secret
chmod 0400 /etc/samba/secret

4) Update fstab to mount the directory

//samba.server.monzell.com/shared /mnt/shared cifs _netdev,credentials=/etc/samba/pw 0 0

5) Finally, mount the share:

mount /mnt/shared

As you can tell, I got it down cold. Why? Because until today, I couldn’t do step 5. I kept getting permission errors:

mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Now I was able to mount if I remove the hosts allow entry:

comment = “shared directory”
path = /srv/directory_to_be_shared
writable = yes
browsable = yes
write list = +shared

But that would mean that I wouldn’t be able to use ACL controls.

After some searching, I found that I can block via IP, which is sort of better – but I still wasn’t satisfied.

I looked at the walkthroughs for all the RHCE books (Van Vugt, Ghori, Jang, Tecmint) and so far, from what I can tell, it should work. I mean, surely the authors have all figured it out, right?

Well, today, I gave it one more and something occur to me that, perhaps, Samba don’t do lookups by default. Sure enough, after some searching, I found:


In order for host allow entries using hostnames to work you need to enable

hostname lookups = yes
In the global configuration of smb.conf.

And sure enough, adding that in smb.conf:

hostname lookups = yes

Allow me to mount with using host controls on the hostname.

Turns out that hostname lookups are quite expensive, resource-wise, so samba have it turned off by default.

I am not sure why all the major RHCE prep books missed this. I thought at first that it may a problem with the editing, which I could understand for one book

But all four?


The simple way of setting up NAT with iptables and Red Hat / CentOS and Scientific Linux

I always feel intimidated with setting NAT. I knew, of course, how to filter packets with iptables, but I always felt that NAT to be just a tad bit confusing. Then I studied for the RHCE exam and it turns out that it is a lot easier than I expected, with a little help from Red Hat’s firewall tool.

With earlier releases of Red Hat, the firewall tool is limited to just filtering by ports with no NAT or ACLs support. With version 6, the tool was significantly revamped. While ACLs are still not supported, you can now setup NAT with the firewall GUI. To get started, log yourself into the X-Windows GUI and run the following from the terminal.


Alternatively, you may run it from the menu -> System- Administration -> firewall. Or if you are logged remotely and you have X-Window running on your workstation, you can forward the tool over SSH to run it locally on your workstation by logging to the server with:

ssh -X username@hostname

and then run the “system-config-firewall.”

(One notable caution: if you have an existing iptables configuration, running this tool will wipe out your rules. Be sure to back /etc/system/iptables before you proceed further on this.)

From there, you will see the following options available:

Select “Masquerading” and then to your right, select the interface or interfaces you want your traffic to go through.

Congratulations, NAT is now setup! The next step is to setup forwarding to your devices or servers. Select “Port Forwarding” to your left:

Then click on add button at your right:


Then near the top of the following screen under the Source heading, click on the interface button, select your interface and click on OK

Then click the protocol button, select your protocol and then click on OK

Then click on the port button and select (or enter) your port(s) and then click on  OK:

Now under the Destination heading, select “Forward to another port” and enter the IP address of the machine you will forward the packets to in the field:

Then select the Port button or enter the port(s) you will be forwarding to:

Click on OK,  then OK to return to the main screen

From there, click on apply (which will write to the /etc/sysconfig/iptables file) and then reload (which will restart iptables).

(If you have backed up your iptables file, you may be able restore them with the custom rules option. However, the iptable backup must be in the iptables-save format, which presumably means that a custom-made one will not work).

You are mostly done here, but there is a couple of more changes youneed to make. First of all, though, lets login in via the command and take a look at the iptables file:

[root@sl6vmware sysconfig]# cat iptables

# Firewall configuration written by system-config-firewall

# Manual customization of this file is not recommended.






-A PREROUTING -i eth+ -p tcp --dport 20:21 -j DNAT --to-destination







-A INPUT -p icmp -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -i eth+ -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT


-A FORWARD -p icmp -j ACCEPT


-A FORWARD -i eth+ -j ACCEPT

-A FORWARD -o eth+ -j ACCEPT

-A FORWARD -i eth+ -m state --state NEW -m tcp -p tcp -d --dport 20:21 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -j REJECT --reject-with icmp-host-prohibited


As you can see there, setting up NAT comprised of a few steps:

1) you set the table type, which at the beginning, would be *nat. If you were to do it from the command line, it would be the first part of the following statement

iptables -t nat <statement>

2) Then, you would set POSTROUTING s. From the command line, it will the second part of the statement we mention earlier.

iptables -t nat -A POSTROUTING -o eth+ -j MASQUERADE

At that point, the nat is setup.

3) From this point, it is just the matter of forwarding the packets based on source port (you can add the ACLs at this point, but the GUI tool will most likely overwrite them. Again, make sure that you have a backup!) . That is done by setting up a PREROUTING rule for a port, with:

-A PREROUTING -i eth+ -p tcp –dport 20:21 -j DNAT –to-destination

From the command line, it would be:

iptables -t nat -A PREROUTING -i eth+ -p tcp –dport 20:21 -j DNAT –to- destination

And then, you would add a FORWARD rule with:

-A FORWARD -i eth+ -m state –state NEW -m tcp -p tcp -d – dport 20:21 -j ACCEPT

Which, from the command line would be:

iptables -t filter -A FORWARD -i eth+ -m state –state NEW -m tcp -p tcp -d –dport 20:21 -j ACCEPT

Note that statement must be under the filter table, not under the nat table.

Getting this right via the command-line is fairly tricky even for experienced administrations, which is why the firewall tool is a great way to manage your rules for NAT (assuming that they are simple enough)

But you are not done yet. First, you need to make sure that you actually enable IP forwarding with the kernel.  You can set it dynamically with either:

sysctl -w net.ipv4.conf.all.forwarding =1


echo “1” > /proc/sys/net/ipv4/ip_forward

To change it permanently (which is what you want), add or change the  net.ipv4.conf.all.forwarding value  in /etc/sysctl.conf to:

net.ipv4.ip_forward = 1

And then reboot (assuming that you didn’t make the change dynamically).

At this point, there is one more thing you may  need to do, , particularly if you are running a FTP service.

With FTP, when a client connects to a server, the FTP server returns the response via different port (usually a high-number port above 1024). That becomes a problem when the client attempts to respond to that same port when the server is behind either a firewall or a NAT , as that port will most likely be blocked.

So there are 4 options you choose to address this problem.

1) Use SFTP. It is is encrypted and travels through the same port (22), so that is the best option. However, not all clients would have thats support.

2) Have the FTP server return a response to a specific port and open that port.

3) Have the client use passive mode

4) Modify the nat/firewall dynamically as well as modify the PORT and PASV commands dynamically.

We will go with the last option. Fortunately, it is not complicated in this case. We just need the open the following file:


And add (or change) the following:


This is a helper module that allows us to dynamically set the ports and track the FTP connectivity. It will load once you restart type iptables, either from the gui or from the command line with:

service iptables restart

If you were to prefer the load the module manual, you would run:

modprobe nf_nat_ftp

Which will load that module as the other supporting modules. You confirm whether it is loaded by running:

lsmod | grep ftp

Which will present you with the following output

nf_nat_ftp              3473  0 

nf_conntrack_ftp       12879  1 nf_nat_ftp

nf_nat                 22788  3 nf_nat_ftp,ipt_MASQUERADE,iptable_nat

nf_conntrack           79611  8 nf_nat_ftp,nf_conntrack_ftp,ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4,nf_conntrack_ipv6,xt_state

Now you are done with NAT. You can start forwarding traffic to your internal servers and have them accessible from outside your network. 

For more information on setting up NAT, go to the following links:

http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch14_:_Linux_Firewalls_Using_iptables – A longer guide to setting up IP tables, with a section for NAT.

http://debianclusters.org/index.php/NAT_with_IPTables – Setting up IP tables with Debian-type distributions.

http://www.ncftp.com/ncftpd/doc/misc/ftp_and_firewalls.html – The problems you will encounter with with NAT and FTP.

A Slightly Longer Guide to Setting up a Caching / Forwarding Name Server

Setting up a caching server on RHEL6, by default, works out of the box for a RHEL-type distributions. From there, it is just the matter of adding additional changes to make it either a forwarding, master or slave name server.

Of course, in our case, there were glitches.

Reviewing through Michael Jang’s RHCE prep book, the instructions for setting up the caching name server goes as following:

1) Install bind-*

2) Enable named with “chkconfig named on”

3) Start up named with “service named start”

That worked.

What did not work was this:

rndc status

That returned with the following message:

  [root@localhost ~]# rndc  status

  rndc: neither /etc/rndc.conf nor /etc/rndc.key was found

As it turns out, rndc.key was not generated when we installed bind. So let’s try to resolve this. While we are on the task, we will go and setup some ACLs and forwarding as well.

First, fixing the immediate problem involves running a command to generate a new key:

  rndc-confgen -a

Which will put in a new rndc.key file by default into /etc.  

However, even though it defaults to a key length of 128, a generating the key on the a virtual machine took a long while. So in the interest of time as well as looking to try something new, we bypassed it by using the OS’s random generator:

  rndc-confgen -r /dev/urandom -a

(Note: you probably wouldn’t want to do this in a mission critical environment).

With the file generated:

  [root@localhost etc]# cat /etc/rndc.key 

  key “rndc-key” {

   algorithm hmac-md5;

  secret “WxQEDQ8KY+SII48TfeV92w==”;


We will next need to give it the appropriate permissions and ownership:

  chown root:named rndc.key 

  chmod 644 rndc.key 

  chcon -u system_u -t etc_t rndc.key

What we have done here is put the group ownership under named group and then make it readable to the named user can read it. Then afterward, we used selinux to give it the appropriate security context for named.

With that done, we insert the following line in at the end of the named.conf file.

  include “/etc/rndc.key”;

We will also need  to setup the forwarders somewhere in the options { } section.

  forwarders {; };

And disable the following lines as well, since we are not using dnssec in our network, otherwise, we will not get a response back for our internal domains:

  # dnssec-enable yes;

  # dnssec-validation yes;

  # dnssec-lookaside auto;

  # bindkeys-file “/etc/named.iscdlv.key”;

(In real life, dnssec is highly recommend for managing DNS in order to avoid DNS poisoning. For now, we’ll work on that another time)

While we are at it, lets add some ACLs to named.conf:

  controls {

         inet allow {localhost;};


Now we restart named and run the original rndc status command:

[root@localhost etc]# rndc status

version: 9.7.3-P3-RedHat-9.7.3-2.el6_1.P3.2

CPUs found: 1

worker threads: 1

number of zones: 19

debug level: 0

xfers running: 0

xfers deferred: 0

soa queries in progress: 0

query logging is OFF

recursive clients: 0/0/1000

tcp clients: 0/100

server is up and running

To ensure that our caching server works, we added the following as the first line in /etc/resolv.conf


If your server is on a DHCP network and you are using dhclient (like mine), you can have added to the resolver file by adding following line in the file dhclient.conf (in my case, /etc/dhcp/dhclient-eth0.conf):

prepend domain-name-servers;

With that done, let’s do a test. We will run the following command to query for an internal domain:

dig @ kvm0001.monzell.com

While on the DNS server, we will monitor DNS traffic from the client (which in this case,

The result? On the client:

[root@localhost data]# dig @ kvm0001.monzell.com

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-2.el6_1.P3.2 <<>> @ kvm0001.monzell.com

; (1 server found)

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45507

;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1


;kvm0001.monzell.com. IN A


kvm0001.monzell.com. 56 IN A


monzell.com. 41 IN NS ns.monzell.com.


ns.monzell.com. 56 IN A

;; Query time: 0 msec


;; WHEN: Sun Sep 11 14:11:56 2011

;; MSG SIZE  rcvd: 86

On the DNS server:

[root@centos log]# tcpdump -i eth0 port 53 and src host

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

14:19:31.219035 IP > centos.monzell.com.domain:  10966+ [1au] A? kvm0001.monzell.com. (48)

So we know that the client has queried the DNS and retrieved the correct IP. Now lets see if the client had it cached now. We run the command the again:

[root@localhost data]# dig @ kvm0001.monzell.com

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-2.el6_1.P3.2 <<>> @ kvm0001.monzell.com

; (1 server found)

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5672

;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1


;kvm0001.monzell.com. IN A


kvm0001.monzell.com. 5 IN A


monzell.com. 5 IN NS ns.monzell.com.


ns.monzell.com. 5 IN A

;; Query time: 0 msec


;; WHEN: Sun Sep 11 14:15:56 2011

;; MSG SIZE  rcvd: 86

Again, we got the correct IP returned. Did it work?

Yes, because on the DNS server, we did not receive query from the client:

  [root@centos log]# tcpdump -i eth0 port 53 and src host

  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

  listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

  14:19:31.219035 IP > centos.monzell.com.domain:  10966+ [1au] A? kvm0001.monzell.com. (48)

  14:21:07.437358 IP 192.16

In fact, we did not see any output until we retried the same dig command a few minutes later:

  [root@centos log]# tcpdump -i eth0 port 53 and src host

  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

  listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

  14:19:31.219035 IP > centos.monzell.com.domain:  10966+ [1au] A? kvm0001.monzell.com. (48)

  14:21:07.437358 IP > centos.monzell.com.domain:  22604+ [1au] A? kvm0001.monzell.com. (48)

At this point, we have verified that caching server works. We are done here.