Deploying SMTP server using Postfix and OpenDKIM

Days ago we got some issue with Amazon SES and decide to make our own SMTP relay service. I tried the combinations of sendmail with dim-milter but for some reason I could not make it work so I start another server from scratch which did work this time. This post mainly focus on configuring OpenDKIM as Postfix is fairly straightforward.

The first thing to do is disabling sendmail which is the default SMTP client on Amazon Linux:

service sendmail stop
chkconfig sendmail off

And then installing Postfix and configuring it:

yum install postfix
cp /etc/postfix/main.cf /etc/postfix/main.cf.original 
vi /etc/postfix/main.cf (appendix 01)

Configuration mainly involves adding DKIM settings (such as milter socket info) and modifying receptions restrictions. In the new setting we define sender_access file to contain the list of senders who can relay through our SMTP service.

cat "DOMAIN.COM OK" >> /etc/postfix/sender_access
postmap /etc/postfix/sender_access

Now we are done with Postfix so it is better to check by sending a test mail.

service postfix start
chkconfig postfix on

The main concern of this post, DKIM, is to cryptographically validate the sender is really from that domain (i.e. domain.com). The validation process starts with the SMTP server signing the email using its private key and then the destination mail server tries to match the private key with the public key obtained from senders’ claimed domain DNS. Once the private and public key matched then it means the SMTP server is from the domain it claims to be.

To install the DKIM we need some API from sendmail and openssl:

yum install sendmail-devel openssl-devel

openDKIM is not available in default repository so we will add it and then install it:

rpm -Uvh http://mirror.pnl.gov/epel/6/i386/epel-release-6-8.noarch.rpm
yum --disablerepo=* --enablerepo=epel install opendkim

Once installation finished we proceed to configurations. There are 3 main files to configure: opendkim.conf contains the main configs such as address of signing table file, key table file and the socket address to listen. The key table file contains the list of keys. The signing table defines which domains should be signed by which key. You will also need to add trusted IP addresses of senders in /etc/opendkim/TrustedHosts to grant them access to SMTP.

cp /etc/opendkim.conf /etc/opendkim.conf.original
vi /etc/opendkim.conf (appendix 02)
vi /etc/opendkim/KeyTable  (appendix 03)
vi /etc/opendkim/SigningTable (appendix 04)

Now we need to generate a pair of keys (private and public) which the public key will be added into DNS records of send domain:

mkdir /etc/opendkim/keys/DOMAIN.COM
opendkim-genkey -D /etc/opendkim/keys/DOMAIN.COM/ -d DOMAIN.COM -s default
mv /etc/opendkim/keys/DOMAIN.COM/default.private /etc/opendkim/keys/DOMAIN.COM/default
chown -R opendkim:opendkim /etc/opendkim/keys/DOMAIN.COM
cat /etc/opendkim/keys/DOMAIN.COM/default.txt

Then start the service:

service opendkim start
chkconfig opendkim on

Finally, you have to add a TXT record in your DNS dashboard. The record name should be default._domainkey.DOMAIN.COM and it should contains something like the following (based on the /etc/opendkim/keys/DOMAIN.COM/default.txt):
“v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHY7Zl+n3SUldTYRUEU1BErHkKN0Ya52gazp1R7FA7vN5RddPxW/sO9JVRLiWg6iAE4hxBp42YKfxOwEnxPADbBuiELKZ2ddxo2aDFAb9U/lp47k45u5i2T1AlEBeurUbdKh7Nypq4lLMXC2FHhezK33BuYR+3L7jxVj7FATylhwIDAQAB”

/etc/postfix/main.cf (appendix 01)

smtpd_milters           = inet:127.0.0.1:2525
non_smtpd_milters       = $smtpd_milters
milter_default_action   = accept

smtpd_error_sleep_time = 1s
smtpd_soft_error_limit = 10
smtpd_hard_error_limit = 20

smtpd_recipient_restrictions = 
	 permit_mynetworks
	 check_sender_access hash:/etc/postfix/sender_access
	 reject_unauth_destination

queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
mail_owner = postfix

inet_interfaces = all
inet_protocols = all

mydestination = $myhostname, localhost.$mydomain, localhost
unknown_local_recipient_reject_code = 550

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases

debug_peer_level = 2
debugger_command =
	 PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
	 ddd $daemon_directory/$process_name $process_id & sleep 5

sendmail_path = /usr/sbin/sendmail.postfix
newaliases_path = /usr/bin/newaliases.postfix
mailq_path = /usr/bin/mailq.postfix
setgid_group = postdrop
html_directory = no

manpage_directory = /usr/share/man
sample_directory = /usr/share/doc/postfix-2.6.6/samples
readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES

/etc/opendkim.conf (appendix 02)

Canonicalization        relaxed/relaxed
ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts
KeyTable                refile:/etc/opendkim/KeyTable
LogWhy                  Yes
MinimumKeyBits          1024
Mode                    sv
PidFile                 /var/run/opendkim/opendkim.pid
SigningTable            refile:/etc/opendkim/SigningTable
Socket                  inet:2525@127.0.0.1
Syslog                  Yes
SyslogSuccess           Yes
TemporaryDirectory      /var/tmp
UMask                   022
UserID                  opendkim:opendkim

/etc/opendkim/KeyTable (appendix 03)

default._domainkey.DOMAIN.COM:default:/etc/opendkim/keys/DOMAIN.COM/default

/etc/opendkim/SigningTable (appendix 04)

*@DOMAIN.COM default._domainkey.DOMAIN.COM

Alternative Python installation in EC2

There are case that we can not just upgrade the python to match applications requirement. EC2 instance uses Python 2.7 and I needed to use Python 3.4. Upgrading the Python might cause unexpected effects on Amazon scripts so I decided to have two Pythons without any conflict.

To start, download the desired version. Enter the directory and follow these commands:

./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make
make altinstall
/usr/local/bin/python3.4 -V

Adding Network Address Translation (NAT) to Amazon private VPC

Assuming that the VPC is ready and there is one public subnet and one private subnet.

Just add an instance (I used Amazon Linux) in public subnet and all incoming/outgoing traffic. It is important to disable source/destination check on that instance (right click on the EC2 instance and you will see it).

Next you need to SSH into the NAT server and run the following commands:

sudo sysctl -q -w net.ipv4.ip_forward=1 net.ipv4.conf.eth0.send_redirects=0
sudo iptables -t nat -A POSTROUTING -o eth0 -s 10.0.0.0/16 -j MASQUERADE

And then test it they are all set:

sudo iptables -n -t nat -L POSTROUTING
sysctl net.ipv4.ip_forward
sysctl net.ipv4.conf.eth0.send_redirects

In the end, go back to AWS console. Go to VPC service and select the route table that is associated with the private network. Then change the default route (0.0.0.0/0) to the NAT instance.

Now you are good to go! All instances in your private subnet have internet access now.

Attaching raw EBS volume to EC2 instance

If you create a volume manually and then try to attach and mount it to an instance you will notice that the volume is not usable cause it is raw. Of course you will need to format it and then mount it.

It is very basic and simple task but not for beginners. So we will list the attached volumes, check the filesystem (which supposed to be nothing!), create file system, create mount point and mount.

lsblk
sudo file -s /dev/xvdb
sudo mkfs -t ext4 /dev/xvdb
sudo mkdir mount_point
sudo mount /dev/xvdb mount_point

Extending Linux physical partition in AWS EC2

Once I got to the problem that an old server migrated to AWS had used 100% of available storage. Lucky me, it was in AWS EC2 and I were able to simply create a snapshot of the EBS volume and then create another volume from the snapshot which had twice the size of old one.

*It is notable that data lost was intolerable!

Until here was easy, and I thought it is going to remain that way using resize2fs tool. But the tool simply said that there partition is already using full disk size! while there was unused space there. So after some search I found the http://litwol.com/content/fdisk-resizegrow-physical-partition-without-losing-data-linodecom article and adapt it to my case.

The difference of my case was that the server partition was DOS partition and not ex* partition. This is what I did to fix it:

List disks and write down “Start” and “Id” for /dev/xvda1 (These two piece of data are extremely important for successfully finishing the task). In my case Start was 63 and Id was 8e.

fidsk -l

And then delete the partition, and create larger a new one with exact same start point:

fdisk -c=dos -u=sectors /dev/xvdc

(The red characters are my response)

(fdisk) Command (m for help): d
(fdisk) Partition number (1-4): 1
(fdisk) Command (m for help): n
(fdisk) Partition type:
p primary
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (63-40558591, default 63): 63
Last sector, +sectors or +size{K,M,G} (63-40558591, default 40558591): 40558591
(fdisk) Command (m for help): t
Partition number (1-4): 1
Hex Code (type L to list codes): 8e
(fdisk) Command (m for help): w

Now we use resize2fs to finalize everything:

resize2fs /dev/xvdc1

At the end you can check the storage device by “lsblk” or “fdisk -l”.

Create custom service to auto start commands on boot

There are many solutions to automatically execute shell commands during the start-up process. My favorite is using services.

So we are going to create a small script in /etc/init.d/ and use chkconfig to auto start it on boot. We call the service “starter”!

vi /etc/init.d/starter

And add the following code to starter:

#!/bin/bash
# chkconfig: 345 99 10
# description: starting some commands at boot
#
case "$1" in
 'start')
	/usr/local/bin/searchd
	python /opt/deploy/main.py
	echo "started" >> /home/user/service.track
	;;
 'stop')
	killall searchd
	killall python
	;;
esac

And finally set permissions and configure the chkconfig:

chmod +x /etc/init.d/starter
chkconfig starter on

Adding custom metric to Amazon Cloud Watch

The default metrics in AWS cloud watch does not include Memory usage or Storage usage data. In my opinion cloud watch has what it takes to monitor a server (and more). So what we are going to do is adding additional metrics to cloud watch.

At first we need to create a user and grant necessary permissions for (we will use the access-key and secret-key of the user):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "cloudwatch:PutMetricData",
        "ec2:DescribeTags"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

Then we need to install the dependencies, download the script and extract it:

sudo yum install -y perl-Switch perl-Sys-Syslog perl-LWP-Protocol-https
wget http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip
unzip CloudWatchMonitoringScripts-v1.1.0.zip
cd aws-scripts-mon

That is it. Done!

To make sure everything is set we do a little test but without sending data to cloud watch:

mon-put-instance-data.pl --mem-util --disk-space-util --disk-path=/ --verbose --aws-access-key-id=youraccesskey --aws-secret-key=yoursecretkey

Now we should add a cron job to send metric in any interval (we used 5 minutes). Enter the “crontab -e” command and add the following:

*/5 * * * * /home/user/aws-scripts-mon/mon-put-instance-data.pl --aws-access-key-id=youraccesskey --aws-secret-key=yoursecretkey --mem-util --disk-space-util --disk-path=/ --from-cron

And restart the crond service

sudo /etc/init.d/crond restart

*It is notable that you can add other metrics too like what we did: –mem-util is for sending memory usage and –disk-space-util is for sending the storage usage.