VALT Warm Standby

⚠DISCLAIMER: This is not a step by step instruction. This article assumes the reader has a working understanding of the VALT application and how to modify the configuration files. This procedure should only be completed by an IVS support engineer. This may be incorporated into the command line menu in the future.

About

The easiest way to achieve a highly available VALT server is by utilizing a VM on a Hypervisor with shared storage that supports High Availability (HA). Both VMWare and Hyper-V have integrated HA capabilities. If there is a need to make the VALT server highly available, IVS strongly recommends deploying the VALT appliance in a virtual environment and leveraging the hypervisor's HA capabilities. If this is not an option, the procedure below will allow you to configure two VALT appliances with one as a primary server and the second as a warm standby. This procedure will configure the MYSQL databases in a master slave configuration and replicate all changes from the primary to the standby server. Video files can be synchronized periodically. The configuration given below will replicate video files on a daily basis, but this can be customized by adjusting the cronjob scheduling.

---

Prerequisites

• A DNS entry (CNAME or A) resolving to the primary server.
  • This should not be the primary DNS entry for that server and will be changed in the event of a failure.
  • If https will be used this DNS entry should be used as the CN for the certificate
    • SANs should be included for the FQDN of each server in addition to the DNS entry
  • For this to work properly there must not be an entry in /etc/hosts for the DNS entry
  • Additionally both servers will need to be pointed at a DNS server that can properly resolve the shared DNS entry.
• /usr/local/WowzaStreamingEngine/conf/dustin/Application.xml should be configured to point to the shared DNS entry.
• The config files in /etc/apache2/sites-enabled/ should reference the shared DNS entry.
• The only config file in /etc/apache2/sites-enabled/ should be valt.conf. If default-ssl.conf is in this directory, delete it.
• If using https, the certificate should already be installed on the primary server

---

Script Package

1. Download VALTRedundancy.tar.gz by running the following command: wget --content-disposition https://ivs.box.com/shared/static/1062styfwr387efew51vt6mzqzsdd1m3.gz -O VALTRedundancy.tar.gz
2. Extract the file using the following command: tar -xvf VALTRedundancy.tar.gz

---

Primary Server Config

Run PrimaryServerConfig.sh on the primary server. Make sure to modify the variables at the top of the script to reflect your environment. Specifically the standbyip must be changed to the ip address of the standby server.

The script will complete by providing the log file name and position. Make a note of these values as they will be needed to configure the standby server.

---

Standby Server Config

1. Run sudo -i

   If you do not use this specific command portions of the script will not execute properly.

2. Run VALTStandbyServerConfig.sh on the standby server. Make sure to modify the variables at the top of the script to reflect your environment. Specifically the primaryip must be changed to the ip address of the primary server, the logfile and logpos variables must be set to the values output when running the script on the primary server.

If using the VALTRedundancy.tar.gz package, the failure.sh and restore.sh scripts should automatically be copied to the /usr/local/valt/bin/ directory and set to executable. These scripts are provided below if the VALTRedundancy.tar.gz package was not used. It is a good idea to update the variables in these scripts now so they are ready to be used in the event of a failover.

---

How to Initiate a Failover to the Standby Server

1. Update the shared DNS entry to reference the standby server.
2. Run the script by typing the following command:

   sudo /usr/local/valt/bin/failure.sh

---

How to Return to the Primary Server

1. Update the shared DNS entry to reference the standby server.
2. Update the variables in the script below to reflect the correct primary ip, slaveuser, and slavepassword. (If this was not done during initial setup)
3. Run the script by typing the following command:

   sudo /usr/local/valt/bin/restore.sh

Scripts

ValtPrimaryServerConfig.sh

#!/bin/bash
# This script will configure an existing VALT server to function as a primary server. 
# You will need to run VALTStandbyServerConfig.sh on the warm standby server.
standbyip="192.168.0.98"
dbuser="ivsadmin"
dbpass="password"
slaveuser="ivs_user"
slavepassword="password"

echo "WARNING: DO NOT RUN THIS SCRIPT MULTIPLE TIMES!!!"
echo "You will need to run VALTStandbyServerConfig.sh on the warm standby server."
echo "Press enter to continue."
read r

if [[ $EUID -ne 0 ]]; then
  	echo "You must run this script as root" 2>&1
  	exit 1
else
	apt install rsync
	echo -e "ivsadmin ALL=(ALL) NOPASSWD: /usr/bin/rsync,/usr/bin/systemd-resolve" >> /etc/sudoers

	# ************************************************************************************
	# The IP address may need to be overridden if the IP was not set through the menu 
	# or if the client interface is not used for replication.
        # ************************************************************************************
	# ipaddress="192.168.0.99"

	if [ -s "/usr/local/valt/conf/ipaddress" ];
	then
		ipaddress=$(</usr/local/valt/conf/ipaddress)
	fi
	sed -i "s|bind-address|#bind-address|g" /etc/mysql/mysql.conf.d/mysqld.cnf
	echo -e "bind-address\t\t= $ipaddress" >> /etc/mysql/mysql.conf.d/mysqld.cnf
	echo -e "server-id\t\t= 1" >> /etc/mysql/mysql.conf.d/mysqld.cnf
	echo -e "log_bin\t\t\t= /var/log/mysql/mysql-bin.log" >> /etc/mysql/mysql.conf.d/mysqld.cnf
	echo -e "binlog_do_db\t\t= v3" >> /etc/mysql/mysql.conf.d/mysqld.cnf
	service mysql restart
	sed -i "s|database_host: localhost|database_host: $ipaddress|g" /var/www/v3/app/config/parameters.yml
	iptables -A INPUT -p tcp -m tcp --dport 3306 -j ACCEPT
	iptables -A OUTPUT -p tcp -m tcp --dport 22 -j ACCEPT
	mysql --defaults-extra-file=/etc/mysql/debian.cnf -e "GRANT ALL PRIVILEGES ON *.* to '$dbuser'@'$standbyip' IDENTIFIED BY '$dbpass';"
	mysql --defaults-extra-file=/etc/mysql/debian.cnf -e "GRANT REPLICATION SLAVE ON *.* TO '$slaveuser'@'$standbyip' IDENTIFIED BY '$slavepassword';"
	mysql --defaults-extra-file=/usr/local/valt/conf/sql.cnf -e "FLUSH PRIVILEGES;"
	mysql --defaults-extra-file=/usr/local/valt/conf/sql.cnf -e "USE v3;"
	mysql --defaults-extra-file=/usr/local/valt/conf/sql.cnf -e "FLUSH TABLES WITH READ LOCK;"
	tmp1=$(mysql --defaults-extra-file=/usr/local/valt/conf/sql.cnf v3 -e "SHOW MASTER STATUS\G;" | grep File)
	sqlfile=${tmp1#*:} 
	tmp1=$(mysql --defaults-extra-file=/usr/local/valt/conf/sql.cnf v3 -e "SHOW MASTER STATUS\G;" | grep Position)
	position=${tmp1#*:}
	sqlfile=$(echo $sqlfile | xargs)
	mysqldump --defaults-extra-file=/usr/local/valt/conf/sql.cnf --opt v3 > /usr/local/valt/backup/v3_sync.sql
	mysql --defaults-extra-file=/usr/local/valt/conf/sql.cnf -e "UNLOCK TABLES;"
	netfilter-persistent save
	echo -e "Master Log File: $sqlfile"
	echo -e "Master Log Position: $position"
fi

ValtStandbyServerConfig.sh

#!/bin/bash
# This script will configure an existing VALT server to function as a 
# standby server.
# You will need to run VALTPrimaryServerConfig.sh on the primary server.

primaryuser="ivsadmin" 
primarypass="password" 
primaryip="192.168.0.99" 
slaveuser="ivs_user"
slavepassword="password"
logfile=""
logpos=""

echo "WARNING: DO NOT RUN THIS SCRIPT MULTIPLE TIMES!!!"
echo "MAKE SURE YOU HAVE UPDATED THE LOGFILE, LOGPOS, AND PRIMARYIP VARIABLES IN THIS SCRIPT!!"
echo "MAKE SURE YOU RAN sudo -i PRIOR TO EXECUTING THIS SCRIPT!!"
echo "Press CTRL-C to cancel or press enter to continue."

read r

if [[ $EUID -ne 0 ]]; then
  	echo "You must run this script as root" 2>&1
  	exit 1
else
	apt install rsync
	ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -N ""
	ssh-copy-id -i /root/.ssh/id_rsa.pub ivsadmin@$primaryip
	rsync --rsync-path="sudo rsync" -avz ivsadmin@$primaryip:/usr/local/WowzaStreamingEngine/content/valt_recordings /usr/local/WowzaStreamingEngine/content/
	(crontab -l ; echo -e '00 11 * * * rsync --rsync-path="sudo rsync" -avz ivsadmin@$primaryip:/usr/local/WowzaStreamingEngine/content/valt_recordings /usr/local/WowzaStreamingEngine/content/') | crontab -
	service apache2 stop
	service WowzaStreamingEngine stop
	service valtcontrols stop
	systemctl disable apache2 
	systemctl disable valtcontrols
	systemctl disable WowzaStreamingEngine
	scp ivsadmin@$primaryip:/usr/local/WowzaStreamingEngine/conf/dustin/Application.xml /usr/local/WowzaStreamingEngine/conf/dustin/Application.xml
	
	#*********************************************************************
	# The section below is for HTTPS enabled environments only
	#*********************************************************************
	mkdir /etc/apache2/ssl
	scp ivsadmin@$primaryip:/etc/apache2/ssl/* /etc/apache2/ssl
	scp ivsadmin@$primaryip:/etc/apache2/sites-enabled/v3.conf /etc/apache2/sites-enabled/
	scp ivsadmin@$primaryip:/usr/local/WowzaStreamingEngine/conf/valtwowza.jks /usr/local/WowzaStreamingEngine/conf/
	scp ivsadmin@$primaryip:/var/www/v3/nodejs/server.js /var/www/v3/nodejs/
	scp ivsadmin@$primaryip:/usr/local/WowzaStreamingEngine/java/lib/security/cacerts /usr/local/WowzaStreamingEngine/java/lib/security/
	scp ivsadmin@$primaryip:/usr/local/WowzaStreamingEngine/conf/VHost.xml /usr/local/WowzaStreamingEngine/conf/
	if ! iptables -S | grep ACCEPT | grep INPUT | grep 443; then
		iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
	fi
	if ! iptables -S | grep ACCEPT | grep INPUT | grep 444; then
		iptables -A INPUT -p tcp -m tcp --dport 444 -j ACCEPT
	fi
	a2enmod ssl
	
	iptables -A OUTPUT -p tcp -m tcp --dport 22 -j ACCEPT
	iptables -A OUTPUT -p tcp -m tcp --dport 3306 -j ACCEPT
	netfilter-persistent save
	cp /usr/local/valt/conf/sql.cnf /usr/local/valt/conf/standby.cnf
	
	echo -e "[client]" > /usr/local/valt/conf/primary.cnf
	echo -e "host=$primaryip" >> /usr/local/valt/conf/primary.cnf
	echo -e "user=$primaryuser" >> /usr/local/valt/conf/primary.cnf
	echo -e "password=$primarypass" >> /usr/local/valt/conf/primary.cnf
	
	mysqldump --defaults-extra-file=/usr/local/valt/conf/standby.cnf --opt v3 > /usr/local/valt/backup/v3_orig.sql
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "DROP DATABASE v3;"
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "CREATE DATABASE v3;"
	scp ivsadmin@$primaryip:/usr/local/valt/backup/v3_sync.sql /usr/local/valt/backup/v3_sync.sql
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf v3 < /usr/local/valt/backup/v3_sync.sql
	echo -e "server-id\t\t= 2" >> /etc/mysql/mysql.conf.d/mysqld.cnf
	echo -e "relay-log\t\t= /var/log/mysql/mysql-relay-bin.log" >> /etc/mysql/mysql.conf.d/mysqld.cnf
	echo -e "log_bin\t\t\t= /var/log/mysql/mysql-bin.log" >> /etc/mysql/mysql.conf.d/mysqld.cnf
	echo -e "binlog_do_db\t\t= v3" >> /etc/mysql/mysql.conf.d/mysqld.cnf
	service mysql restart
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "CHANGE MASTER TO MASTER_HOST='$primaryip',MASTER_USER='$slaveuser', MASTER_PASSWORD='$slavepassword', MASTER_LOG_FILE='$logfile', MASTER_LOG_POS=  $logpos;"
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "START SLAVE;"
	cp failure.sh /usr/local/valt/bin/
	cp restore.sh /usr/local/valt/bin/
	chmod +x /usr/local/valt/bin/failure.sh
	chmod +x /usr/local/valt/bin/restore.sh
fi

failure.sh

#!/bin/bash
#This script will make the redundant VALT server the primary VALT server. 
#The DNS entry must be updated to point to this server.

echo "WARNING: DO NOT RUN THIS SCRIPT MULTIPLE TIMES!!!"
echo "MAKE SURE THE DNS ENTRY HAS BEEN UPDATED PRIOR TO EXECUTING THIS SCRIPT"
echo "Press enter to continue."
read r


if [[ $EUID -ne 0 ]]; then
        echo "You must run this script as root" 2>&1
        exit 1
else
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "STOP SLAVE;"
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "RESET SLAVE;"
	systemd-resolve --flush-caches
	systemctl enable apache2 
	systemctl enable valtcontrols
	systemctl enable WowzaStreamingEngine
	service apache2 start
	service WowzaStreamingEngine start
	service valtcontrols start
fi

restore.sh

#!/bin/bash
# This script will migrate the database and all recordings back to the primary VALT server. 
# The DNS entry must be updated to point to the primary VALT server.

primaryip="192.168.0.99"
slaveuser="ivs_user"
slavepassword="password"

echo "WARNING: DO NOT RUN THIS SCRIPT MULTIPLE TIMES!!!"
echo "MAKE SURE THE DNS ENTRY HAS BEEN UPDATED PRIOR TO EXECUTING THIS SCRIPT"
echo "Press enter to continue."
read r
export NCURSES_NO_UTF8_ACS=1 
if [[ $EUID -ne 0 ]]; then
  	echo "You must run this script as root" 2>&1
  	exit 1
else
	systemd-resolve --flush-caches
	service apache2 stop
	service WowzaStreamingEngine stop
	service valtcontrols stop
	systemctl disable apache2 
	systemctl disable valtcontrols
	systemctl disable WowzaStreamingEngine
	rsync --rsync-path="sudo rsync" -avz /usr/local/WowzaStreamingEngine/content/valt_recordings ivsadmin@10.111.3.73:/usr/local/WowzaStreamingEngine/content/
	mysqldump --defaults-extra-file=/usr/local/valt/conf/standby.cnf --opt v3 > /usr/local/valt/backup/v3.sql
	mysql --defaults-extra-file=/usr/local/valt/conf/primary.cnf -e "DROP DATABASE v3;"
	mysql --defaults-extra-file=/usr/local/valt/conf/primary.cnf -e "CREATE DATABASE v3;"
	mysql --defaults-extra-file=/usr/local/valt/conf/primary.cnf v3 < /usr/local/valt/backup/v3.sql
	mysql --defaults-extra-file=/usr/local/valt/conf/primary.cnf v3 -e "FLUSH TABLES WITH READ LOCK;"
	tmp1=$(mysql --defaults-extra-file=/usr/local/valt/conf/primary.cnf v3 -e "SHOW MASTER STATUS\G;" | grep File)
	sqlfile=${tmp1#*:}
	tmp1=$(mysql --defaults-extra-file=/usr/local/valt/conf/primary.cnf v3 -e "SHOW MASTER STATUS\G;" | grep Position)
	position=${tmp1#*:}
	sqlfile=$(echo $sqlfile | xargs)
	mysqldump --defaults-extra-file=/usr/local/valt/conf/primary.cnf --opt v3 > /usr/local/valt/backup/v3_sync.sql
	mysql --defaults-extra-file=/usr/local/valt/conf/primary.cnf v3 -e "UNLOCK TABLES;"
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "DROP DATABASE v3;"
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "CREATE DATABASE v3;"
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf v3 < /usr/local/valt/backup/v3_sync.sql
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "CHANGE MASTER TO MASTER_HOST="\'"$primaryip"\'", MASTER_USER="\'"$slaveuser"\'", MASTER_PASSWORD="\'"$slavepassword"\'", MASTER_LOG_FILE="\'"$sqlfile"\'", MASTER_LOG_POS = $position;"
	mysql --defaults-extra-file=/usr/local/valt/conf/standby.cnf -e "START SLAVE;"
fi