Kubernetes Bare Metal Walkthrough

The purpose of this guide is to setup a simple Kubernetes cluster with one control-plane node (or “master” node), and one worker node (or “minion” node) on CentOS. This guide is not intended to be used for Production, however, it could be used to get started on building a Production cluster. This guide also assumes that you are installing the Kuberenetes cluster on freshly installed hosts with no previous existing software’s and configurations. You should have your networking and hostnames configured, SELinux diabled, and for this simple dev cluster, should have the IP and hostname mapping already setup in /etc/hosts.

You will need to turn swap off on your hosts. Kubelet will not work otherwise.

swapoff -a
sed -i '/swap/d' /etc/fstab

Setup the Docker repo for each host:

sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

Install Docker on each host:

sudo yum -y install docker

sudo systemctl start docker
sudo systemctl enable docker

We are now ready to being installing Kubernetes.

Setup the Kubernetes repo on each host:

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Install the following packages on each host:

yum install -y kubelet kubeadm kubectl

Start Kubelet service on each host

systemctl enable kubelet
systemctl start kubelet

On the master node, set the following firewall rules:

firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=2379-2380/tcp
firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=10251/tcp
firewall-cmd --permanent --add-port=10252/tcp
firewall-cmd --permanent --add-port=10255/tcp
firewall-cmd --reload

On the worker node, set the following firewall rules:

firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=10251/tcp
firewall-cmd --permanent --add-port=10255/tcp
firewall-cmd --reload

On each host, set the net.bridge.bridge-nf-call-iptables to ‘1’ in your sysctl config file so that packets are properly processed by iptables for filtering and port forwarding:

cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF


sysctl --system

On the master node, initialize a cluster with pod network CIDR of 10.244.0.0/16. This is required by flannel:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

*** IMPORTANT *** Make note of the last line of output (kubeadm join …). You will need to run this later.

On master node, to start using the cluster you need to run it as a regular user by typing:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Set Up Pod Network
A Pod Network allows nodes within the cluster to communicate. We’re using flannel for this purpose.

On the master node, install flannel with the command:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Wait a a few seconds, and then on master node, confirm everything is “ready” or “running”. If not, check back a minute later.

kubectl get nodes
kubectl get pods --all-namespaces

On your worker nodes, run the “kubeadm join” line you got from the “kudeadm init” earlier.

Flush iptables on all nodes

sudo -s
systemctl stop kubelet
systemctl stop docker
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start docker

echo "iptables --flush" >> /etc/rc.d/rc.local
echo "iptables -tnat --flush" >> /etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local
systemctl enable rc-local
exit

Back to the master, check and make sure status’ are “Ready”

kubectl get nodes -o wide

 

You have now successfully setup a simple Kubernetes Bare Metal cluster.

Expanding a GPT partitioned volume in Linux

The GPT partition table is both at the beginning and end of the disk. When you expand your volume, you will need to re-write the GPT backup headers at the end of the expanded volume.

 

Enter into the “parted” utility on your resized volume.

$ parted /dev/nvmeX

 

Try to print the partition table and you will get a warning message. This is where you will re-write the GPT backup headers. Tell the parted utility to “fix” the backup GPT table.

(parted) p                                                                

Error: The backup GPT table is not at the end of the disk, as it should be.  This might mean that another operating system believes the disk is smaller.  Fix, by moving the backup to the end (and removing the old backup)?

Fix/Ignore/Cancel? Fix                                                    

Warning: Not all of the space available to /dev/nvme3n1 appears to be used, you can fix the GPT to use all of the space (an extra 419430400 blocks) or continue with the current setting? 

Fix/Ignore? Fix

 

Print your partition table, and you will see it list.

(parted) p  

 

Now you are able to continue with resizing partitions or adding new ones to your volume.

DCCP Linux Kernel local privilege escalation vulnerability (CVE-2017-6074)

A vulnerability has been found in the DCCP Linux kernel module which allows a local, unprivileged user to escalate privileges on a Linux system. DCCP is used to manage network traffic congestion in the application layer.

This issue affects Red Hat and CentOS releases 5, 6, and 7, as well as other Linux distributions. You should update your Kernel as soon as possible.

To mitigate without a Kernel upgrade, run the following command and reboot your system:

echo "install dccp /bin/true" >> /etc/modprobe.d/disable-dccp.conf

This will disable the DCCP module from loading on boot.

More information can be found here on Red Hat’s website.

Red Hat 7 Get back eth Network Device Names

On installation of Red Hat or CentOS 7, boot the Anaconda installer with net.ifnames=0 parameter.

On installer boot, hit the esc key.

anaconda-1

On the “boot” prompt, enter linux net.ifnames=0 and hit enter. The installer will now boot.

anaconda-2

Go into the network settings in the installer, and you should now see your network devices named eth0, eth1, etc.

anaconda-3

Dirty COW Red Hat/CentOS patches released

Patches to fix the Dirty COW vulnerability have been released by Red Hat and CentOS for RHEL/CentOS. The patch is presented as a kernel upgrade and should be applied as soon as possible. You can read more about this patch on Red Hat’s website.

To patch your server, simply run the following command:

yum install kernel

After the kernel is installed, reboot your server. Once your server comes back online, you can confirm the patched kernel is now running with the following command:

uname -r

For RHEL/CentOS 6 systems, you should see “2.6.32-642.6.2.el6” from the output of the command.

For RHEL/CentOS 7 systems, you should see “3.10.0-327.36.3.el7” from the output of the command.

Linux audit “Backlog limit exceeded”

If you’re running a busy Linux system, you may see the following error in your Kernel logs:
“audit: backlog limit exceeded”.

For example:
Linux audit "Backlog limit exceeded" 1

To alleviate the message output in your logs, you can increase the audit buffer.

Edit /etc/audit/audit.rules and increase the value for “-b”. For Red Hat Linux 6 systems, the default value is 320.
Linux audit "Backlog limit exceeded" 2

Determining the appropriate value may require some time and experimentation. As a general rule, we suggest doubling the value and then observing it’s affects. It is recommended not to set the value too high, as it may cause increased system resource usage.

Once your value is set, save the file and restart the auditd service.
Linux audit "Backlog limit exceeded" 3

Please note that the “audit: backlog limit exceeded” message is a generic message and could be a symptom of a bigger issue (most common, log writing issues due to ext4 file system issues). Further troubleshooting may be necessary.

MySQL 5.6 “innodb_table_stats” not found

Fix this annoying error in MySQL 5.6

On systems that were upgraded from MySQL 5.5 to 5.6 (Oracle, Percona, etc.) – you may see the following error:

Error: Table “mysql”.”innodb_table_stats” not found.

Here’s a more complete example:

2016-06-14 15:08:03 7f82b3fff700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2016-06-14 15:08:03 7f82b3fff700 InnoDB: Recalculation of persistent statistics requested for table "mydb1"."cron_schedule" but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.
2016-06-14 15:08:04 7f82b3fff700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.
2016-06-14 15:08:04 7f82b3fff700 InnoDB: Recalculation of persistent statistics requested for table "mydb2"."cron_schedule" but the required persistent statistics storage is not present or is corrupted. Using transient stats instead.

Why is this error occuring?

MySQL 5.6 introduces a new “persistent optimizer statistics” feature which stores statistical data about your databases in the “mysql” database. These tables do not get created when upgrading from MySQL 5.5 to 5.6, which causes heavy error logging in mysqld.log.

How to fix the innodb_table_stats not found error

To fix this issue, you need to manually create the tables:

DROP TABLE IF EXISTS `innodb_index_stats`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `innodb_index_stats` (
 `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
 `table_name` varchar(64) COLLATE utf8_bin NOT NULL,
 `index_name` varchar(64) COLLATE utf8_bin NOT NULL,
 `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
 `stat_name` varchar(64) COLLATE utf8_bin NOT NULL,
 `stat_value` bigint(20) unsigned NOT NULL,
 `sample_size` bigint(20) unsigned DEFAULT NULL,
 `stat_description` varchar(1024) COLLATE utf8_bin NOT NULL,
 PRIMARY KEY (`database_name`,`table_name`,`index_name`,`stat_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0;
/*!40101 SET character_set_client = @saved_cs_client */;

DROP TABLE IF EXISTS `innodb_table_stats`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `innodb_table_stats` (
 `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
 `table_name` varchar(64) COLLATE utf8_bin NOT NULL,
 `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
 `n_rows` bigint(20) unsigned NOT NULL,
 `clustered_index_size` bigint(20) unsigned NOT NULL,
 `sum_of_other_index_sizes` bigint(20) unsigned NOT NULL,
 PRIMARY KEY (`database_name`,`table_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0;
/*!40101 SET character_set_client = @saved_cs_client */;

DROP TABLE IF EXISTS `slave_master_info`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `slave_master_info` (
 `Number_of_lines` int(10) unsigned NOT NULL COMMENT 'Number of lines in the file.',
 `Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log currently being read from the master.',
 `Master_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last read event.',
 `Host` char(64) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT 'The host name of the master.',
 `User_name` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The user name used to connect to the master.',
 `User_password` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The password used to connect to the master.',
 `Port` int(10) unsigned NOT NULL COMMENT 'The network port used to connect to the master.',
 `Connect_retry` int(10) unsigned NOT NULL COMMENT 'The period (in seconds) that the slave will wait before trying to reconnect to the master.',
 `Enabled_ssl` tinyint(1) NOT NULL COMMENT 'Indicates whether the server supports SSL connections.',
 `Ssl_ca` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Authority (CA) certificate.',
 `Ssl_capath` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path to the Certificate Authority (CA) certificates.',
 `Ssl_cert` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL certificate file.',
 `Ssl_cipher` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the cipher in use for the SSL connection.',
 `Ssl_key` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL key file.',
 `Ssl_verify_server_cert` tinyint(1) NOT NULL COMMENT 'Whether to verify the server certificate.',
 `Heartbeat` float NOT NULL,
 `Bind` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'Displays which interface is employed when connecting to the MySQL server',
 `Ignored_server_ids` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The number of server IDs to be ignored, followed by the actual server IDs',
 `Uuid` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The master server uuid.',
 `Retry_count` bigint(20) unsigned NOT NULL COMMENT 'Number of reconnect attempts, to the master, before giving up.',
 `Ssl_crl` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Revocation List (CRL)',
 `Ssl_crlpath` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path used for Certificate Revocation List (CRL) files',
 `Enabled_auto_position` tinyint(1) NOT NULL COMMENT 'Indicates whether GTIDs will be used to retrieve events from the master.',
 PRIMARY KEY (`Host`,`Port`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Master Information';
/*!40101 SET character_set_client = @saved_cs_client */;

DROP TABLE IF EXISTS `slave_relay_log_info`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `slave_relay_log_info` (
 `Number_of_lines` int(10) unsigned NOT NULL COMMENT 'Number of lines in the file or rows in the table. Used to version table definitions.',
 `Relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the current relay log file.',
 `Relay_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The relay log position of the last executed event.',
 `Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log file from which the events in the relay log file were read.',
 `Master_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last executed event.',
 `Sql_delay` int(11) NOT NULL COMMENT 'The number of seconds that the slave must lag behind the master.',
 `Number_of_workers` int(10) unsigned NOT NULL,
 `Id` int(10) unsigned NOT NULL COMMENT 'Internal Id that uniquely identifies this record.',
 PRIMARY KEY (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Relay Log Information';
/*!40101 SET character_set_client = @saved_cs_client */;

DROP TABLE IF EXISTS `slave_worker_info`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `slave_worker_info` (
 `Id` int(10) unsigned NOT NULL,
 `Relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
 `Relay_log_pos` bigint(20) unsigned NOT NULL,
 `Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
 `Master_log_pos` bigint(20) unsigned NOT NULL,
 `Checkpoint_relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
 `Checkpoint_relay_log_pos` bigint(20) unsigned NOT NULL,
 `Checkpoint_master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
 `Checkpoint_master_log_pos` bigint(20) unsigned NOT NULL,
 `Checkpoint_seqno` int(10) unsigned NOT NULL,
 `Checkpoint_group_size` int(10) unsigned NOT NULL,
 `Checkpoint_group_bitmap` blob NOT NULL,
 PRIMARY KEY (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Worker Information';
/*!40101 SET character_set_client = @saved_cs_client */;

As soon as these tables are created, MySQL should start storing data in these tables and the error logging about “innodb_table_stats not found” should stop.

Red Hat Enterprise Linux 6.8 Released

10-04_6_2_redhat_logo
Red Hat has released update 8 of RHEL 6. Read the press release here.

Along with security updates, this release features the ability to expand the XFS file system to up 300TB, and the ability to create a deployable snapshot of your running system. You can find out more about this technology, called Relax-and-Recover, here.

More detailed information about this release can be found in the release notes here.