AdvertPro is designed to work best with CentOS and Red Hat Enterprise Linux. All of our development, testing and support staff training is done with CentOS. For this reason, we strongly recommend that you choose one of these supported Linux distributions. In fact, these instructions presume that you will be taking our advice!
Need help deciding what type of server or how many servers to get? Check out our system administrators guide for some guidance. There is also our clustering guide which explains how to scale out AdvertPro and handle failover. If you still have questions, please contact technical support and they'll be happy to guide you in the right direction.
Tip
Downloading and installing Nginx is as easy as running the following command:
yum install nginx
You may, however, need to download and install the Extra Packages for Enterprise Linux repository configuration (EPEL) if you get an error saying that Nginx is not available.
rpm -Uvh http://mirrors.kernel.org/fedora-epel/6/x86_64/epel-release-6-8.noarch.rpm
Of course this only provides a basic Nginx installation which is not quite what we need. Integrating the ey_balancer module with Nginx requires compiling a new Nginx binary from source. The source of the ey_balancer module also needs a minor patch to work with Nginx 1.4.x versions. To reduce errors in these steps, we strongly recommend that you download this already patched Nginx 1.4.6 bundle that we've prepared:
To be able to compile Nginx from source you are also going to need GCC, make and a couple of other required libraries for regex and SSL support, which can also be installed with yum:
yum install gcc
yum install make
yum install openssl-devel
yum install openssh-clients
yum install pcre-devel
Downloading and installing MySQL is as easy as running the following command:
yum install mysql mysql-server
MariaDB is also supported provided that you use the MyISAM storage engine. In the future we may support the newer Aria storage engine. However, at the time of writing we do not recommend using Aria in a production environment as it has not been tested sufficiently.
Another option is TokuDB, which we highly recommend for customers with high-traffic deployments. In fact, you can use TokuDB with either MySQL or MariaDB. Please make sure that you are using AdvertPro 4.0 or newer versions, however, as previous versions are not fully optimized for the TokuDB storage engine. It's also strongly recommended that you use SSD disks with TokuDB, preferably in a RAID10 configuration with 4-6 disks to get the best possible performance.
The following steps are necessary to compile and install Nginx with the ey_balancer module. We also include the http_ssl module for SSL support. All other modules that are not necessary for the operation of AdvertPro have been disabled. This results in a faster binary that is better hardened for security as well.
tar -xzvf nginx-1.4.6-eybalancer.tar.gz
cd nginx-1.4.6
./configure --user=nginx --group=nginx --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --with-http_ssl_module --without-http_auth_basic_module --without-http_autoindex_module --without-http_charset_module --without-http_empty_gif_module --without-http_fastcgi_module --without-http_geo_module --without-http_limit_req_module --without-http_limit_conn_module --without-http_map_module --without-http_memcached_module --without-http_referer_module --without-http_scgi_module --without-http_split_clients_module --without-http_ssi_module --without-http_userid_module --without-http_uwsgi_module --add-module=../ngx_max_connections-0.0.5 --with-file-aio --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=-Wl,-E
make
cp objs/nginx /usr/sbin/nginx_eybalancer
rm -f /etc/nginx/conf.d/*.conf
rm -f /etc/nginx/fastcgi*
rm -f /etc/nginx/koi-*
rm -f /etc/nginx/mime.types.default
rm -f /etc/nginx/nginx.conf.default
rm -f /etc/nginx/scgi*
rm -f /etc/nginx/uwsgi*
rm -f /etc/nginx/win-utf
mkdir -p /var/lib/nginx/cache
After the compilation is done you will need to edit the /etc/rc.d/init.d/nginx startup file and replace the definitions of the nginx and prog variables with the following:
nginx="/usr/sbin/nginx_eybalancer"
prog="nginx"
This facilitates starting the custom nginx_eybalancer binary rather than the nginx binary that was installed from EPEL, which lacks the ey_balancer module.
To be safe, we also suggest that you disable new versions of Nginx from being installed automatically via yum update by editing the /etc/yum.conf file and adding the following exclusion rule in the [main] section:
exclude=nginx*
This prevents newer versions of Nginx from being installed automatically and possibly changing the startup script back to the EPEL provided binary without your knowledge!
rpm -i jdk-{version}-linux-i586.rpm
rpm -i jdk-{version}-linux-x64.rpm
mv apache-tomcat-{version}.tar.gz /usr/local
cd /usr/local
tar -xzvf apache-tomcat-{version}.tar.gz
mv apache-tomcat-{version} tomcat
rm -rf apache-tomcat-{version}.tar.gz
cd tomcat
chmod 755 bin/*.sh
cd webapps
rm -rf *
mkdir ROOT
yum install apr-devel
yum install gcc
yum install make
yum install zlib-devel
cd /usr/local/tomcat/bin
tar -xzf tomcat-native.tar.gz
cd tomcat-native*
cd jni/native
./configure --with-java-home=/usr/java/latest --with-apr=/usr/bin/apr-1-config --with-ssl=no
make
make install
unzip javamail-{version}.zip
cp javamail-{version}/mail.jar /usr/local/tomcat/lib/mail.jar
unzip jaf-{version}.zip
cp jaf-{version}/activation.jar /usr/local/tomcat/lib/activation.jar
tar -xzvf mysql-connector-java-{version}.tar.gz
cd mysql-connector-java-{version}
cp mysql-connector-java-{version}-bin.jar /usr/local/tomcat/lib/mysql.jar
cp advertpro-{version}.war /usr/local/tomcat/webapps/ROOT
cd /usr/local/tomcat/webapps/ROOT
/usr/java/latest/bin/jar -xf advertpro-{version}.war
rm -f advertpro-{version}.war
The following environment variables should be added to the /etc/profile file.
export JAVA_HOME=/usr/java/latest
export PATH=$PATH:$JAVA_HOME/bin
You either need to log out and log back in or run those export commands at this point, which will add those environment variables to your shell session.
Replace the entire contents of the /etc/my.cnf with the following recommended configuration:
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
datadir = /var/lib/mysql
character-set-server=utf8
skip-external-locking
max_connections = 384
key_buffer_size = 64M
table_open_cache = 256
thread_cache_size = 8
bulk_insert_buffer_size = 16M
join_buffer_size = 1M
read_buffer_size = 2M
read_rnd_buffer_size = 2M
sort_buffer_size = 4M
max_allowed_packet = 16M
max_heap_table_size = 32M
tmp_table_size = 32M
query_alloc_block_size = 16384
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1
query_prealloc_size = 16384
myisam_sort_buffer_size = 64M
open-files-limit=8192
server-id = 1
symbolic-links = 0
[mysql.server]
basedir = /var/lib
user = mysql
[mysqld_safe]
err-log = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid
[mysqldump]
max_allowed_packet = 16M
quick
[mysql]
default-character-set=utf8
no-auto-rehash
[isamchk]
key_buffer = 128M
read_buffer = 2M
write_buffer = 2M
sort_buffer = 128M
[myisamchk]
key_buffer = 128M
read_buffer = 2M
write_buffer = 2M
sort_buffer = 128M
[mysqlhotcopy]
interactive-timeout
Note
Replace the entire contents of the /etc/nginx/nginx.conf file with the following recommended configuration:
#---------------------------------------------------------------------
# Runtime Settings
#---------------------------------------------------------------------
user nginx;
pid /var/run/nginx.pid;
error_log /var/log/nginx/error.log crit;
#---------------------------------------------------------------------
# Performance Settings
#---------------------------------------------------------------------
worker_processes 32; # change this to match number of CPU cores
worker_rlimit_nofile 32768;
events {
worker_connections 8192; # (32*(8192/4)) = 65,536 max clients
}
#---------------------------------------------------------------------
# HTTP Settings
#---------------------------------------------------------------------
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log off;
gzip on;
gzip_comp_level 6;
gzip_http_version 1.1;
gzip_min_length 0;
gzip_types application/json application/x-javascript text/plain text/css text/javascript text/xml;
gzip_vary on;
sendfile on;
tcp_nopush on;
keepalive_timeout 60;
keepalive_disable msie6;
proxy_cache_path /var/lib/nginx/cache levels=1:1:2 keys_zone=shared:100m inactive=60m max_size=2500m;
proxy_cache_use_stale updating;
server_tokens off;
upstream tomcat {
server 127.0.0.1:8080;
max_connections 256; # set to (tomcat_max_threads / nginx_worker_processes)
max_connections_queue_timeout 30000;
}
server {
listen 80;
server_name _;
location / {
proxy_pass http://tomcat;
proxy_connect_timeout 30;
proxy_set_header Host $http_host;
proxy_set_header Accept-Encoding "";
proxy_set_header User-Agent $http_user_agent;
proxy_set_header Referer $http_referer;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect http://$http_host $scheme://$http_host;
}
location /servlet/files {
proxy_pass http://tomcat;
proxy_set_header Host $http_host;
proxy_cache shared;
proxy_cache_key "$scheme$http_host$request_uri";
}
location ~* \.(css|html|js|txt|xml)$ {
proxy_pass http://tomcat;
proxy_set_header Host $http_host;
proxy_set_header Accept-Encoding "";
proxy_cache shared;
proxy_cache_key "$scheme$http_accept_encoding$http_host$request_uri";
}
location ~* \.(gif|jpg|png|jar)$ {
proxy_pass http://tomcat;
proxy_cache shared;
proxy_cache_key "$scheme$http_host$request_uri";
}
}
}
Note
Edit /usr/local/tomcat/bin/catalina.sh and add the following settings after the comments/instructions at the top.
DO NOT JUST COPY THESE SETTINGS
Read the instructions contained within the settings to properly set the -Xmx, -Xms, -XX:NewSize, -XX:MaxNewSize and -XX:ParallelGCThreads options -- failure to do so will likely result in degraded performance.
# Tomcat Home.
CATALINA_HOME="/usr/local/tomcat"
# Java Home.
JAVA_HOME="/usr/java/latest"
#
# Change -Xms and -Xms if necessary to allocate more or less memory.
# It is important that they have the same value for optimal performance.
# Note that on some operating systems you may need to increase the value of
# -Xss to 512k or higher (do not exceed 1024k) if you experience stability
# problems or start seeing stack overflow exceptions in your application
# server logs.
#
# If you've changed -Xms and -Xms, you should change -XX:NewSize and
# -XX:MaxNewSize to have values equal to half of the -Xms and -Xmx
# values.
#
# Suggested values based on available physical server memory (RAM):
#
# 2 GB RAM -Xms1024m -Xmx1024m -XX:NewSize=512m -XX:MaxNewSize=512m
# 4 GB RAM -Xms2048m -Xmx2048m -XX:NewSize=1024m -XX:MaxNewSize=1024m
# 8 GB RAM -Xms4096m -Xmx4096m -XX:NewSize=2048m -XX:MaxNewSize=2048m
# 16 GB RAM -Xms8192m -Xmx8192m -XX:NewSize=4096m -XX:MaxNewSize=4096m
# 32 GB RAM -Xms16384m -Xmx16384m -XX:NewSize=8192m -XX:MaxNewSize=8192m
#
CATALINA_OPTS="$CATALINA_OPTS -server -Xms512m -Xmx512m -Xss256k -XX:+DisableExplicitGC"
CATALINA_OPTS="$CATALINA_OPTS -XX:NewSize=256m -XX:MaxNewSize=256m"
#
# If you're using a multi processor system, uncomment the following
# options and set the value of -XX:ParallelGCThreads equal to the
# number of physical CPU cores that are in your system.
#
CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParNewGC -XX:ParallelGCThreads=2"
CATALINA_OPTS="$CATALINA_OPTS -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80"
CATALINA_OPTS="$CATALINA_OPTS -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0"
#
# APR Connector JNI Library
#
CATALINA_OPTS="$CATALINA_OPTS -Djava.library.path=/usr/local/apr/lib"
#
# Uncomment the following options to help debug garbage collection
# related performance problems.
#
CATALINA_OPTS="$CATALINA_OPTS -verbose:gc -Xloggc:/usr/local/tomcat/logs/gc.txt -XX:+PrintGCDetails"
#
# Java Options.
#
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=28800 -Dsun.net.inetaddr.negative.ttl=60"
#
# Raise open file limit and stack size.
#
ulimit -n 524288
ulimit -s 2048
Edit the /usr/local/tomcat/conf/logging.properties file and change the following log handlers from INFO to SEVERE level:
1catalina.org.apache.juli.FileHandler.level = SEVERE
2localhost.org.apache.juli.FileHandler.level = SEVERE
3manager.org.apache.juli.FileHandler.level = SEVERE
4host-manager.org.apache.juli.FileHandler.level = SEVERE
java.util.logging.ConsoleHandler.level = SEVERE
Replace the entire contents of the /usr/local/tomcat/conf/server.xml with the following recommended configuration:
<?xml version="1.0" encoding="utf-8"?>
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector protocol="org.apache.coyote.http11.Http11AprProtocol"
acceptCount="128"
connectionTimeout="15000"
disableUploadTimeout="false"
enableLookups="false"
maxKeepAliveRequests="100"
keepAliveTimeout="30000"
maxThreads="8192"
port="8080"
pollTime="2000"
pollerSize="32768"
useSendfile="true"
sendfileSize="1024"
URIEncoding="UTF-8"/>
<Engine defaultHost="ads.yoursite.com" name="Catalina">
<Host name="ads.yoursite.com" appBase="webapps" unpackWARs="true">
<Context docBase="ROOT" path="" privileged="false" swallowOutput="true" />
</Host>
<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>
</Engine>
</Service>
</Server>
Create /etc/rc.d/init.d/tomcat and place the following content in it:
#!/bin/bash
#
# Startup script for the Tomcat Web Server
#
# chkconfig: 345 92 16
# description: Tomcat is a World Wide Web server. It is used to serve \
# HTML, JSP, and servlets, and CGI if needed.
# processname: java
CATALINA_HOME=/usr/local/tomcat
case "$1" in
start)
rm -rf $CATALINA_HOME/work
mkdir $CATALINA_HOME/work
$CATALINA_HOME/bin/startup.sh
;;
restart)
$CATALINA_HOME/bin/shutdown.sh
sleep 30
killall -9 java
sleep 2
rm -rf $CATALINA_HOME/work
mkdir $CATALINA_HOME/work
$CATALINA_HOME/bin/startup.sh
;;
stop)
$CATALINA_HOME/bin/shutdown.sh
sleep 30
killall -9 java
sleep 2
;;
*)
echo $"Usage: tomcat {start|restart|stop}"
exit 1
esac
exit 0
The key thing here is that MySQL should start at level 2 while Nginx and Tomcat should start at level 3 so that MySQL is started before Nginx and Tomcat start.
chmod 755 /etc/rc.d/init.d/mysqld
/sbin/chkconfig --add mysqld
/sbin/chkconfig --level 2345 mysqld on
/sbin/chkconfig --list mysqld
chmod 755 /etc/rc.d/init.d/nginx
/sbin/chkconfig --add nginx
/sbin/chkconfig --level 345 nginx on
/sbin/chkconfig --list nginx
chmod 755 /etc/rc.d/init.d/tomcat
/sbin/chkconfig --add tomcat
/sbin/chkconfig --level 345 tomcat on
/sbin/chkconfig --list tomcat
AdvertPro needs the following ports to be open in your firewall in order to function properly:
| Port Number | Protocol | Allow Input | Allow Output | Client/Source Address | Destination Address | Used By |
|---|---|---|---|---|---|---|
| 25 | TCP | Yes | Yes | Any | Any | SMTP |
| 80 | TCP | Yes | Yes | Any | Any | HTTP |
| 443 | TCP | Yes | Yes | Any | Any | HTTPS |
| 3306 | TCP | Yes | Yes | localhost | localhost | MySQL |
| 9000 | TCP | No | Yes | localhost | websvc1.advertpro.com websvc2.advertpro.com | AdvertPro |
| 9001 | TCP | No | Yes | localhost | websvc1.advertpro.com websvc2.advertpro.com | AdvertPro |
| 9002 | TCP | No | Yes | localhost | websvc1.advertpro.com websvc2.advertpro.com | AdvertPro |
| 9003 | TCP | No | Yes | localhost | websvc1.advertpro.com websvc2.advertpro.com | AdvertPro |
| 9004 | TCP | No | Yes | localhost | websvc1.advertpro.com websvc2.advertpro.com | AdvertPro |
| 9005 | TCP | No | Yes | localhost | websvc1.advertpro.com websvc2.advertpro.com | AdvertPro |
Assuming you will be using the iptables firewall, simply replace the entire contents of the /etc/sysconfig/iptables file with the following:
# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
After making that change you will need to restart iptables, which can be done with the following command:
/etc/rc.d/init.d/iptables restart
Handling a huge number of connections may not work smoothly out of the box. You generally have to do some optimizations to the TCP/IP settings. Start with the following settings and optimize further if needed, but in most cases they'll do fine as-is.
First, you can hot-deploy the following optimized TCP/IP settings by running these commands:
/sbin/sysctl -w net.ipv4.tcp_max_syn_backlog=4096
/sbin/sysctl -w net.ipv4.tcp_syncookies=1
/sbin/sysctl -w net.ipv4.tcp_synack_retries=5
/sbin/sysctl -w net.ipv4.tcp_fin_timeout=15
/sbin/sysctl -w net.core.rmem_default=104448
/sbin/sysctl -w net.core.wmem_default=104448
/sbin/sysctl -w net.core.wmem_max=12582912
/sbin/sysctl -w net.core.rmem_max=12582912
/sbin/sysctl -w net.ipv4.tcp_rmem="10240 87380 12582912"
/sbin/sysctl -w net.ipv4.tcp_wmem="10240 87380 12582912"
/sbin/sysctl -w net.ipv4.tcp_mem="8388608 8388608 8388608"
/sbin/sysctl -w net.core.netdev_max_backlog=2500
/sbin/sysctl -w fs.file-max=6315922
/sbin/sysctl -w net.netfilter.nf_conntrack_max=33554432
/sbin/sysctl -w net.ipv4.tcp_moderate_rcvbuf=1
/sbin/sysctl -w net.ipv4.tcp_window_scaling=1
Then to make these changes permanent the following settings can be added to the end of your /etc/sysctl.conf file:
# Controls the use of TCP syncookies
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 5
# Reduce FIN_WAIT state time
net.ipv4.tcp_fin_timeout = 15
# Larger TCP memory
net.core.rmem_default = 104448
net.core.wmem_default = 104448
net.core.wmem_max = 12582912
net.core.rmem_max = 12582912
net.ipv4.tcp_rmem = 10240 87380 12582912
net.ipv4.tcp_wmem = 10240 87380 12582912
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.core.netdev_max_backlog = 2500
fs.file-max = 6315922
# Larger Connection Table
net.netfilter.nf_conntrack_max = 33554432
# Misc TCP adjustments
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_window_scaling = 1
/etc/rc.d/init.d/mysqld start
/etc/rc.d/init.d/nginx start
/etc/rc.d/init.d/tomcat start
Log in to the MySQL client and execute the following SQL queries to create the AdvertPro database and user to access it.
mysql -u root
CREATE DATABASE advertpro CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON advertpro.* TO advertpro@"localhost" IDENTIFIED BY 'changeme';
GRANT ALL PRIVILEGES ON advertpro.* TO advertpro@"localhost.localdomain" IDENTIFIED BY 'changeme';
FLUSH PRIVILEGES;
If you have purchased a subscription to the GeoIP Country or GeoIP City database, you should install that prior to running the AdvertPro setup utility.