Version 3.0

AdvertPRO - Nginx + ey_balancer + Tomcat - Linux Installation

Prerequisites

These instructions absolutely assume that you are working with a Linux distribution that has yum support, which includes the popular CentOS and Red Hat Enterprise Linux.

These instructions also assume that you'll be deploying on a large server. The reference configuration files we provide assume a server with the equivalent of 32 CPU cores and 64 GB of RAM. If this seems outrageous you probably do not need to be installing Nginx! We only recommend this configuration for clients that will be serving billions of impressions per month.

Step #1: Download Files

AdvertPRO

  1. Open a browser window to: https://services.renegadeinternet.com/download/
  2. Enter your Customer ID and E-mail Address into the form.
  3. Select the operating system your server is running from the Platform list box.
  4. Select the Install package for a clean installation from the Package list box.
  5. Click on the Download Package button to proceed.
  6. Check your e-mail and you should receive an e-mail with a download link within a few minutes.

Tip Tip

If you download the install package on Windows, depending upon the browser you use, the file may be renamed from advertpro-{version}.war to advertpro-{version}.zip since WAR files are essentially ZIP files so you may need to rename the file back to advertpro-{version}.war after downloading if this occurs!

Nginx + ey_balancer

Downloading and installing Nginx is as easy as running the following command:


yum install nginx

You may, however, need to download and install the Extra Packages for Enterprise Linux repository configuration (EPEL) if you get an error saying that Nginx is not available.


rpm -Uvh http://mirrors.kernel.org/fedora-epel/6/x86_64/epel-release-6-8.noarch.rpm

Of course this only provides a basic Nginx installation which is not quite what we need. Integrating the ey_balancer module with Nginx requires compiling a new Nginx binary from source. The source of the ey_balancer module also needs a minor patch to work with Nginx 1.0.x versions. To reduce errors in these steps, we strongly recommend that you download this already patched Nginx 1.0.14 bundle that we've prepared:

To be able to compile Nginx from source you are also going to need GCC, make and a couple of other required libraries for regex and SSL support, which can also be installed with yum:


yum install gcc
yum install make
yum install openssl-devel
yum install openssh-clients
yum install pcre-devel

Java Development Kit

  1. Open a browser window to: http://www.oracle.com/technetwork/java/javase/downloads/
  2. Click on the Java Download graphic for Java Platform (JDK).
  3. Click on the Accept License Agreement radio button on the next page.
  4. Then click on the RPM download link for Linux x86 if you have a 32-bit system or Linux x64 if you have a 64-bit system.

Note Note

JDK 1.4, 1.5 (5.0) and 1.6 (6.0) are also supported, however we recommend using JDK 7.0 (1.7.x) for new installations.

Java Libraries

  1. Download the JavaMail 1.4 library from http://www.oracle.com/technetwork/java/javamail-1-4-140512.html.
  2. Download the JAF 1.1 library from http://www.oracle.com/technetwork/java/jaf11-139815.html.

MySQL Database Server

Downloading and installing MySQL is as easy as running the following command:


yum install mysql mysql-server

MariaDB is also supported provided that you use the MyISAM storage engine. In the future we may support the newer Aria storage engine. However, at the time of writing we do not recommend using Aria in a production environment as it has not been tested sufficiently.

Another option is TokuDB, which we highly recommend for customers with high-traffic deployments. In fact, you can use TokuDB with either MySQL or MariaDB. Please make sure that you are using AdvertPro 3.0 or newer versions, however, as previous versions are not fully optimized for the TokuDB storage engine.

Note Note

We fully support the MySQL 4.0, 4.1, 5.0 and 5.1 series, however we recommend using MySQL 5.5 for new installations.

MySQL Connector/J JDBC Driver

  1. Open a browser window to one of the following, depending on your chosen version of MySQL:
  2. Download the TAR Archive package.

Apache Tomcat Server

  1. Open a browser window to: http://tomcat.apache.org/download-70.cgi
  2. Scroll down to the listing for Tomcat 7.0.x.
  3. Download the Binary Distribution labeled Core: tar.gz.

Step #2: Installation

Nginx + ey_balancer

The following steps are necessary to compile and install Nginx with the ey_balancer module. We also include the http_ssl module for SSL support. All other modules that are not necessary for the operation of AdvertPro have been disabled. This results in a faster binary that is better hardened for security as well.


tar -xzvf nginx-1.0.14-eybalancer.tar.gz
cd nginx-1.0.14
./configure --user=nginx --group=nginx --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --with-http_ssl_module --without-http_auth_basic_module --without-http_autoindex_module --without-http_charset_module --without-http_empty_gif_module --without-http_fastcgi_module --without-http_geo_module --without-http_limit_req_module --without-http_limit_zone_module --without-http_map_module --without-http_memcached_module --without-http_referer_module --without-http_scgi_module --without-http_split_clients_module --without-http_ssi_module --without-http_userid_module --without-http_uwsgi_module --add-module=../ngx_max_connections-0.0.5 --with-file-aio --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=-Wl,-E
make
cp objs/nginx /usr/sbin/nginx_eybalancer
rm -f /etc/nginx/conf.d/*.conf
rm -f /etc/nginx/fastcgi*
rm -f /etc/nginx/koi-*
rm -f /etc/nginx/mime.types.default
rm -f /etc/nginx/nginx.conf.default
rm -f /etc/nginx/scgi*
rm -f /etc/nginx/uwsgi*
rm -f /etc/nginx/win-utf
mkdir -p /var/lib/nginx/cache

After the compilation is done you will need to edit the /etc/rc.d/init.d/nginx startup file and replace the definitions of the nginx and prog variables with the following:


nginx="/usr/sbin/nginx_eybalancer"
prog="nginx"

This facilitates starting the custom nginx_eybalancer binary rather than the nginx binary that was installed from EPEL, which lacks the ey_balancer module.

To be safe, we also suggest that you disable new versions of Nginx from being installed automatically via yum update by editing the /etc/yum.repos.d/epel.repo file and appending the following line to it:


exclude=nginx,nginx*

This prevents newer versions of Nginx from being installed automatically and possibly changing the startup script back to the EPEL provided binary without your knowledge!

Java Development Kit

32-bit


rpm -i jdk-{version}-linux-i586.rpm

64-bit


rpm -i jdk-{version}-linux-x64.rpm

Note Note

You will find more complete installation instructions here: http://download.oracle.com/javase/7/docs/webnotes/install/linux/linux-jdk.html

Apache Tomcat Server


mv apache-tomcat-{version}.tar.gz /usr/local
cd /usr/local
tar -xzvf apache-tomcat-{version}.tar.gz
mv apache-tomcat-{version} tomcat
rm -rf apache-tomcat-{version}.tar.gz
cd tomcat
chmod 755 bin/*.sh
cd webapps
rm -rf *
mkdir ROOT

Apache Tomcat APR Library


yum install apr-devel
yum install zlib-devel
cd /usr/local/tomcat/bin
tar -xzf tomcat-native.tar.gz
cd tomcat-native*
cd jni/native
./configure --with-java-home=/usr/java/latest --with-apr=/usr/bin/apr-1-config --with-ssl=no
make
make install

Java Libraries


unzip javamail-{version}.zip
cp javamail-{version}/mail.jar /usr/local/tomcat/lib/mail.jar
unzip jaf-{version}.zip
cp jaf-{version}/activation.jar /usr/local/tomcat/lib/activation.jar

MySQL Connector/J JDBC Driver


tar -xzvf mysql-connector-java-{version}.tar.gz
cd mysql-connector-java-{version}
cp mysql-connector-java-{version}-bin.jar /usr/local/tomcat/lib/mysql.jar

AdvertPRO


cp advertpro-{version}.war /usr/local/tomcat/webapps/ROOT
cd /usr/local/tomcat/webapps/ROOT
/usr/java/latest/bin/jar -xf advertpro-{version}.war
rm -f advertpro-{version}.war

Step #3: Configuration

Environment Variables

Edit the /etc/profile file and append the following to it to set the JAVA_HOME environment and add the various JDK utilities to your path:


export JAVA_HOME=/usr/java/latest
export PATH=$PATH:$JAVA_HOME/bin

MySQL Settings

Replace the entire contents of the /etc/my.cnf file with the following recommended configuration:


[client]
port = 3306
socket = /var/lib/mysql/mysql.sock

[mysqld]
skip-name-resolve
port = 3306
socket = /var/lib/mysql/mysql.sock
datadir = /var/lib/mysql
default-character-set=utf8
skip-external-locking
max_connections = 4096
key_buffer_size = 6144M
table_open_cache = 8192
thread_cache_size = 128
bulk_insert_buffer_size = 16M
join_buffer_size = 2M
read_buffer_size = 4M
read_rnd_buffer_size = 4M
sort_buffer_size = 16M
max_allowed_packet = 16M
max_heap_table_size = 768M
tmp_table_size = 768M
query_alloc_block_size = 16384
query_cache_limit = 2M
query_cache_size = 768M
query_cache_type = 1
query_prealloc_size = 16384
myisam_sort_buffer_size = 256M
server-id = 1
symbolic-links = 0

[mysql.server]
basedir = /var/lib
user = mysql

[mysqld_safe]
err-log = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid

[mysqldump]
max_allowed_packet = 16M
quick

[mysql]
default-character-set=utf8
no-auto-rehash

[isamchk]
key_buffer = 256M
read_buffer = 2M
write_buffer = 2M
sort_buffer = 256M

[myisamchk]
key_buffer = 256M
read_buffer = 2M
write_buffer = 2M
sort_buffer = 256M

[mysqlhotcopy]
interactive-timeout
        

Note Note

These settings are for MySQL 5.5.x, which is the version we recommend using for new installations. However, they are also compatible with and can be used with the MySQL 5.1.x, 5.0.x and 4.1.x versions. If you are using an even older version of MySQL, please contact technical support for configuration guidance if needed.

Nginx Settings

Replace the entire contents of the /etc/nginx/nginx.conf file with the following recommended configuration:


#---------------------------------------------------------------------
# Runtime Settings
#---------------------------------------------------------------------

user nginx;

pid /var/run/nginx.pid;

error_log /var/log/nginx/error.log crit;

#---------------------------------------------------------------------
# Performance Settings
#---------------------------------------------------------------------

worker_processes 32;    # change this to match number of CPU cores

worker_rlimit_nofile 32768;

events {
  worker_connections 8192;    # (32*(8192/4)) = 65,536 max clients
}


#---------------------------------------------------------------------
# HTTP Settings
#---------------------------------------------------------------------

http {
  include /etc/nginx/mime.types;

  default_type application/octet-stream;

  access_log off;

  gzip on;
  gzip_comp_level 6;
  gzip_http_version 1.1;
  gzip_min_length 0;
  gzip_types application/json application/x-javascript text/plain text/css text/javascript text/xml;
  gzip_vary on;

  sendfile on;
  tcp_nopush on;

  keepalive_timeout 60;

  proxy_cache_path /var/lib/nginx/cache levels=1:1:2 keys_zone=shared:100m inactive=60m max_size=2500m;
  proxy_cache_use_stale updating;

  server_tokens off;

  upstream tomcat {
    server 127.0.0.1:8009;
    max_connections 256;    # set to (tomcat_max_threads / nginx_worker_processes)
    max_connections_queue_timeout 30000;
  }

  server {
    listen 80;

    server_name _;

    location / {
      proxy_pass http://tomcat;

      proxy_connect_timeout 30;

      proxy_set_header Host $http_host;

      proxy_set_header Accept-Encoding "";
      proxy_set_header User-Agent $http_user_agent;
      proxy_set_header Referer $http_referer;

      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $http_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_redirect http://$http_host $scheme://$http_host;
    }

    location /servlet/files {
      proxy_pass http://tomcat;
      proxy_set_header Host $http_host;
      proxy_cache shared;
      proxy_cache_key "$scheme$http_host$request_uri";
    }

    location ~* \.(css|html|js|txt|xml)$ {
      proxy_pass http://tomcat;
      proxy_set_header Host $http_host;
      proxy_set_header Accept-Encoding "";
      proxy_cache shared;
      proxy_cache_key "$scheme$http_accept_encoding$http_host$request_uri";
    }

    location ~* \.(gif|jpg|png|jar)$ {
      proxy_pass http://tomcat;
      proxy_cache shared;
      proxy_cache_key "$scheme$http_host$request_uri";
    }
  }
}

Note Note

Be sure to change the worker_processes, worker_connections and max_connections settings as recommended in the comments to suit your needs. The default settings we've used assume that you will deploy Tomcat to be able to handle 8,192 concurrent clients. Nginx, however, will be able to handle up to 65,536 concurrent clients while the ey_balancer module ensures not to send more than 8,192 concurrent connections to Tomcat. It is important to note that Nginx caches as much content as possible from Tomcat so many requests don't even need to connect to Tomcat. All of the CPU heavy operations like slow client I/O, file serving, GZIP compression and SSL encryption are offloaded to Nginx so in most cases the Tomcat requests take no more than 10-15ms and Nginx is rarely waiting on Tomcat.

Apache Tomcat Environment

Edit /usr/local/tomcat/bin/catalina.sh and add the following after the comments/instructions at the top.


# Tomcat Home.
CATALINA_HOME="/usr/local/tomcat"

# Java Home.
JAVA_HOME="/usr/java/latest"

#
# Change -Xms and -Xms if necessary to allocate more or less memory.
# It is important that they have the same value for optimal performance.
# Note that on some operating systems you may need to increase the value of
# -Xss to 512k or higher (do not exceed 1024k) if you experience stability
# problems or start seeing stack overflow exceptions in your application
# server logs.
#
# If you've changed -Xms and -Xms, you should change -XX:NewSize and
# -XX:MaxNewSize to have values equal to half of the -Xms and -Xmx
# values.
#
# Suggested values based on available physical server memory (RAM):
#
#  2 GB RAM    -Xms1024m     -Xmx1024m     -XX:NewSize=512m     -XX:MaxNewSize=512m
#  4 GB RAM    -Xms2048m     -Xmx2048m     -XX:NewSize=1024m    -XX:MaxNewSize=1024m
#  8 GB RAM    -Xms4096m     -Xmx4096m     -XX:NewSize=2048m    -XX:MaxNewSize=2048m
# 16 GB RAM    -Xms8192m     -Xmx8192m     -XX:NewSize=4096m    -XX:MaxNewSize=4096m
# 32 GB RAM    -Xms16384m    -Xmx16384m    -XX:NewSize=8192m    -XX:MaxNewSize=8192m
# 64 GB RAM    -Xms32768m    -Xmx32768m    -XX:NewSize=16384m   -XX:MaxNewSize=16384m
#
CATALINA_OPTS="$CATALINA_OPTS -server -Xms32768m -Xmx32768m -Xss256k -XX:+DisableExplicitGC"
CATALINA_OPTS="$CATALINA_OPTS -XX:NewSize=16384m -XX:MaxNewSize=16384m"

#
# If you're using a multi processor system, uncomment the following
# options and set the value of -XX:ParallelGCThreads equal to the
# number of processors that are in your system.
#
#CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParNewGC -XX:ParallelGCThreads=32"
#CATALINA_OPTS="$CATALINA_OPTS -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80"
#CATALINA_OPTS="$CATALINA_OPTS -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0"

#
# APR Connector JNI Library
#
CATALINA_OPTS="$CATALINA_OPTS -Djava.library.path=/usr/local/apr/lib"

#
# Uncomment the following options to help debug garbage collection
# related performance problems.
#
#CATALINA_OPTS="$CATALINA_OPTS -verbose:gc -Xloggc:/usr/local/tomcat/logs/gc.txt -XX:+PrintGCDetails"

#
# Java Options.
#
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=28800 -Dsun.net.inetaddr.negative.ttl=60"

#
# Raise open file limit and stack size.
#
ulimit -n 524288
ulimit -s 2048

Apache Tomcat Settings

Edit the /usr/local/tomcat/conf/logging.properties file and change the following log handlers from INFO to SEVERE level:


1catalina.org.apache.juli.FileHandler.level = SEVERE
2localhost.org.apache.juli.FileHandler.level = SEVERE
3manager.org.apache.juli.FileHandler.level = SEVERE
4host-manager.org.apache.juli.FileHandler.level = SEVERE
java.util.logging.ConsoleHandler.level = SEVERE

Replace the entire contents of the /usr/local/tomcat/conf/server.xml with the following recommended configuration:


<?xml version="1.0" encoding="utf-8"?>

<Server port="8005" shutdown="SHUTDOWN">

  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />
  <Listener className="org.apache.catalina.core.JasperListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <Service name="Catalina">

    <Connector protocol="org.apache.coyote.http11.Http11AprProtocol"
               acceptCount="128"
               connectionTimeout="15000"
               disableUploadTimeout="false"
               enableLookups="false"
               maxKeepAliveRequests="100"
               keepAliveTimeout="30000"
               maxThreads="8192"
               port="8009"
               pollTime="2000"
               pollerSize="32768"
               useSendfile="true"
               sendfileSize="1024"
               URIEncoding="UTF-8"/>

    <Engine defaultHost="ads.yoursite.com" name="Catalina">

      <Host name="ads.yoursite.com" appBase="webapps" unpackWARs="true">

        <Context docBase="ROOT" path="" privileged="false" swallowOutput="true" />

      </Host>

      <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase" />

    </Engine>

  </Service>

</Server>
        

Note Note

These settings are for Tomcat 7.0.x, which is the version we recommend using. If you are using an older version of Tomcat, please contact technical support for configuration guidance if needed.

Apache Tomcat Startup

Create /etc/rc.d/init.d/tomcat using the following recommended settings.


#!/bin/bash
#
# Startup script for the Tomcat Web Server
#
# chkconfig: 345 92 16
# description: Tomcat is a World Wide Web server.  It is used to serve \
#              HTML, JSP, and servlets, and CGI if needed.
# processname: java

CATALINA_HOME=/usr/local/tomcat

case "$1" in
  start)
        rm -rf $CATALINA_HOME/work
        mkdir $CATALINA_HOME/work
        $CATALINA_HOME/bin/startup.sh
        ;;
  restart)
        $CATALINA_HOME/bin/shutdown.sh
        sleep 45
        killall -9 java
        sleep 2
        rm -rf $CATALINA_HOME/work
        mkdir $CATALINA_HOME/work
        $CATALINA_HOME/bin/startup.sh
        ;;
  stop)
        $CATALINA_HOME/bin/shutdown.sh
        sleep 45
        killall -9 java
        sleep 2
        ;;
  *)
        echo $"Usage: tomcat {start|restart|stop}"
        exit 1
esac

exit 0
        

Run Level Configuration


chmod 755 /etc/rc.d/init.d/mysqld
/sbin/chkconfig --add mysqld
/sbin/chkconfig --level 2345 mysqld on
/sbin/chkconfig --list mysqld
chmod 755 /etc/rc.d/init.d/nginx
/sbin/chkconfig --add nginx
/sbin/chkconfig --level 2345 nginx on
/sbin/chkconfig --list nginx
chmod 755 /etc/rc.d/init.d/tomcat
/sbin/chkconfig --add tomcat
/sbin/chkconfig --level 345 tomcat on
/sbin/chkconfig --list tomcat

Firewall Configuration

AdvertPRO needs the following ports to be open in your firewall in order to function properly:

Port NumberProtocolAllow InputAllow OutputClient/Source AddressDestination AddressUsed By
25TCPYesYesAnyAnySMTP
80TCPYesYesAnyAnyHTTP
443TCPYesYesAnyAnyHTTPS
3306TCPYesYeslocalhostlocalhostMySQL
9000TCPNoYeslocalhostwebsvc1.advertpro.com
websvc2.advertpro.com
AdvertPRO
9001TCPNoYeslocalhostwebsvc1.advertpro.com
websvc2.advertpro.com
AdvertPRO
9002TCPNoYeslocalhostwebsvc1.advertpro.com
websvc2.advertpro.com
AdvertPRO
9003TCPNoYeslocalhostwebsvc1.advertpro.com
websvc2.advertpro.com
AdvertPRO
9004TCPNoYeslocalhostwebsvc1.advertpro.com
websvc2.advertpro.com
AdvertPRO
9005TCPNoYeslocalhostwebsvc1.advertpro.com
websvc2.advertpro.com
AdvertPRO

Failure to open the aforementioned ports in your firewall will cause the AdvertPRO setup utility to abort.

Step #4: Startup

MySQL Database Server


/etc/rc.d/init.d/mysql start

Apache Tomcat Server


/etc/rc.d/init.d/tomcat start

Nginx Web Server


/etc/rc.d/init.d/nginx start

Step #5: Finishing Up

MySQL Database

Log in to the MySQL client of your choosing and execute the following SQL queries to create the AdvertPRO database and user to access it.


CREATE DATABASE advertpro;
GRANT ALL PRIVILEGES ON advertpro.* TO advertpro@"localhost" IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON advertpro.* TO advertpro@"localhost.localdomain" IDENTIFIED BY 'password';
FLUSH PRIVILEGES;

Warning Warning

It is strongly recommended that you grant all privileges except for Grant to the database. However, if desired the Create_view, Show_view, Create_routine, Alter_routine, and Execute privileges are unused and may be revoked. Do not revoke any other privileges as doing so will prevent the normal operation of AdvertPRO!

Keep in mind that the database will be created using the default MySQL character set, which is specified by the character_set_system option in the my.cnf configuration file. For older versions of MySQL the option is named character_set instead. By default MySQL uses the Latin1 character set, so if you wish to use UTF-8 to support non-Latin1 characters create the MySQL database using this command unless you already changed the default MySQL character set to UTF-8:


CREATE DATABASE advertpro CHARACTER SET utf8 COLLATE utf8_general_ci;

When the AdvertPRO setup utility asks you for the JDBC URL you should changed it to specify that you want to use UTF-8 unless again you already changed the default MySQL character set to UTF-8 as the driver will always use the default MySQL character set regardless of the character set of your database:


jdbc:mysql://localhost:3306/advertpro?useUnicode=true&characterEncoding=UTF-8

GeoIP Database

If you have purchased the GeoIP Country or GeoIP City database, you should install that prior to running the AdvertPRO setup utility.

  1. GeoIP Country Installation
  2. GeoIP City Installation

There are also free versions of GeoIP Country and City that you can download for evaluation purposes, but you should bear in mind that they are not as accurate as the paid versions!

AdvertPRO Setup

  1. Open a browser window to: http://ads.yoursite.com/servlet/setup
  2. Follow the setup utility instructions.
  3. Make sure to follow the post-install instructions given by the setup utility too!

Note Note

It may take about 30 seconds from the time you start Apache Tomcat for it to automatically deploy AdvertPRO the first time.