What is Puppet

Posted: 4p in Linux

What is Puppet 

Puppet is a software which is using for system automation and managemnet. It manages your servers, your described machine configurations in an easy-to-read declarative language, and will bring your systems into the desired state and keep them there.Before talking more about puppet I want to refresh your thoughts about automation. The product is owned by puppetlabs Inc the leader in IT automation .

What is Automation ?

System automation is the use or introduction of automatic configurations , scripts or other process to perform the daily task automaticaly.

 Why Automation ?

  •  Speed           : It will help us to complete the tasks in less time
  •  consistency   : It will avoid human errors which may occur during the repetition
  •  Easy            :  Free from hazards and avoid boredom of repetition

What to Automate ?

Since the servers a infrastructure consist of a certain complexity and valuable data it will not be a wise decision if we choose a wrong thing to automate. So we have to consider few things before start with automation.

Choose the right thing to automate
  • Freaquency : How often we have to perform the task. If the task comes very rare the effort to make those thing automated will be a waist.
  • Variability  : How much similar the tasks are more similar more easy to automate

Dont Learn two things at a time

If we try to automate a technolgy or process in which we are not sounded enough. It will be very defficult to isolate the errors when things go wrong.That means we cant identify the exact issue, whther the issue is with the process we are doing or its with the puppet configurations.

Platform Support

Puppet will work on all operating systems but the pupppet master should be in linux . Windows machines can’t act as puppet master servers. Before installing any Windows agent nodes, be sure that you have a *nix puppet master installed and configured.

Comparison between Puppet and Chef:

 

Puppet

Chef

Definition Puppet is an open source configuration management tool which is written in Ruby. It is a product of Puppet Labs. Chef is also a configuration tool, but it is written in Ruby and Erlang. It is a product of Opscode.
Supported Platforms It is officially supported on a broader range of OS. It is officially supported on a less broader range of OS.
Community Larger user base Comparatively smaller user base
Pricing It has a free open source version. Puppet Enterprise is free for the first 10 nodes and then $99 per node (per year) after that. It also has a free open source version. Private Chef ranges from $120 per month for 20 servers to $600 per month for 100 servers.
API Integration It seems to have no extended API Chef has an extended API
Type of application It is a user application It is also a user application but also can become a part of the application
Configuring the Configuration Server Comparatively difficult Comparatively easy
Code Execution Both on puppermaster and puppet client On the node/client
Ordered Execution Some support Better support
Company Puppet Labs Opscode
Notable Customers Twitter and Nokia Facebook and Splunk
Frindliness More sysadmin friendly More programmer friendly
Language Mainly Puppet’s custom JSON-like language, although a Ruby option is available beginning in version 2.6 A subset of Ruby
Advertisements

Nagios Directory Structure

Main Config File
/usr/local/nagios/etc/nagios.cfg

Log File
/usr/local/nagios/var/nagios.log

Object Config Files
/usr/local/nagios/etc/objects/*.cfg

Nagios Plugins
/usr/local/nagios/libexec     // defined in /usr/local/nagios/etc/resource.cfg file

Nagios Web interface
/usr/local/nagios/etc/cgi.cfg

Nagios config file for Apache to interpret
/usr/local/apache/conf.d/nagios.conf

This contains directives for the following URLs
http://<nagios-host>/nagios/
http://<nagios-host>/nagios/cgi-bin/

Nagios Log rotation configuration File
/etc/logrotate.d/nagios

What is Difference between event, worker and prefork

Apache (HTPD) is  very popular and widely deployed web server arround the world. A-Patchy server comes with multiple modules. The term MPM is used for multiprocessing module. We can check for default mpm by running this command “ httpd -l ”

Apache 2 is available with following 3 MPM modules.

PREFORK
WORKER

EVENT

(mpm_winnt This Multi-Processing Module is optimized for Windows NT.)
(mpm_netware Multi-Processing Module implementing an exclusively threaded web server optimized for Novell NetWare)

Prefork MPM

A prefork mpm handles http requests just like older Apache 1.3. As the name suggests it will pre-fork necessary child process while starting Apache. It is suitable for all those websites which don’t want threading for compatibility. i.e for non-thread-safe libraries . It is also known as the best MPM for isolating each incoming http request.

How it works

A single control (master) process is responsible for launching multiple child processes which serves incoming http requests. Apache always tries to maintain several spare (not-in-use) server processes, which stand ready to serve incoming requests. In this way, clients do not need to wait for a new child processes to be forked before their requests can be served.
We can adjust this spare process through the Apache configuration. Default settings are usually enough for small amount of traffic. One can always tune those Directives / Values as per their requirements.

Pre-Fork is the default module given by Apache.

#prefork MPM
#StartServers: number of server processes to start
#MinSpareServers: minimum number of server processes which are kept spare
#MaxSpareServers: maximum number of server processes which are kept spare
#ServerLimit: maximum value for MaxClients for the lifetime of the server
#MaxClients: maximum number of server processes allowed to start
#MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

Worker MPM

A worker mpm is an Multi-Processing Module (MPM) which implements a hybrid multi-process multi-threaded server. By using threads to serve requests, it is able to serve a large number of requests with fewer system resources than a process-based server.

The most important directives used to control this MPM are ThreadsPerChild, which controls the number of threads deployed by each child process and MaxClients, which controls the maximum total number of threads that may be launched.

Strength : Memory usage and performance wise its better than prefork
Weakness : worker will not work properly with languages like php

How it works

A single control process (the parent) is responsible for launching child processes. Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, as well as a listener thread which listens for connections and passes them to a server thread for processing when they arrive.

Apache always tries to maintain a group of spare or idle server threads, which stand ready to serve incoming requests. In this way, clients do not need to wait for a new threads or processes to be created before their requests can be served. The number of processes that will initially launched is set by the StartServers directive. During operation, Apache assesses the total number of idle threads in all processes, and forks or kills processes to keep this number within the boundaries specified by MinSpareThreads and MaxSpareThreads. Since this process is very self-regulating, it is rarely necessary to modify these directives from their default values. The maximum number of clients that may be served simultaneously (i.e., the maximum total number of threads in all processes) is determined by the MaxClients directive. The maximum number of active child processes is determined by the MaxClients directive divided by the ThreadsPerChild directive

#worker MPM
#StartServers: initial number of server processes to start
#MaxClients: maximum number of simultaneous client connections
#MinSpareThreads: minimum number of worker threads which are kept spare
#MaxSpareThreads: maximum number of worker threads which are kept spare
#ThreadsPerChild: constant number of worker threads in each server process
#MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule worker.c>
StartServers         4
MaxClients         300
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

Event MPM

The event Multi-Processing Module (MPM) is designed to allow more requests to be served simultaneously by passing off some processing work to supporting threads, freeing up the main threads to work on new requests. Event has been released with stable in Apache 2.4. The Event MPM works the exact same way as the Worker MPM when it comes to processes and threads. The big difference is that an Event MPM will dedicate a thread to a request, not the whole HTTP connection.

How it works

This MPM tries to fix the ‘keep alive problem’ in HTTP. After a client completes the first request, the client can keep the connection open, and send further requests using the same socket. This can save significant overhead in creating TCP connections. However, Apache HTTP Server traditionally keeps an entire child process/thread waiting for data from the client, which brings its own disadvantages. To solve this problem, this MPM uses a dedicated thread to handle both the Listening sockets, all sockets that are in a Keep Alive state, and sockets where the handler and protocol filters have done their work and the only remaining thing to do is send the data to the client. The status page of mod_status shows how many connections are in the mentioned states.

This is useful in a situation where you like the idea of threading, but have an application that uses rather long KeepAlive timeouts. With the Worker MPM, the thread would be bound to the connection, and stayed tied up regardless if a request was being processed or not.

With the Event MPM, the connection the thread is only used for requests and frees backup immediately after the request is fulfilled, regardless of the actual HTTP connection, which is handled by the parent process.  Since the thread frees up immediately after the request is fulfilled,  it can be used for other requests.

<IfModule event.c>
MinSpareThreads 64
MaxSpareThreads 128
ThreadsPerChild 64
ThreadLimit 64
MaxRequestsPerChild 20000
ListenBacklog 4096
</IfModule>

What is initrd image in Linux

The initial RAM disk (initrd) is an initial root file system that is mounted prior to when the real root file system is available. The initrd is bound to the kernel and loaded as part of the kernel boot procedure. The kernel then mounts this initrd as part of the two-stage boot process to load the modules to make the real file systems available and get at the real root file system.

initrd provides the capability to load a RAM disk by the boot loader. This RAM disk can then be mounted as the root filesystem and programs can be run from it. Afterwards, a new root file system can be mounted from a different device. The previous root (from initrd) is then moved to a directory and can be subsequently unmounted.
How initrd works

initrd provides the capability to load a RAM disk by the bootloader. This RAM disk can then be mounted as the root fileystem and programs can be run from it. Afterwards, a new root file system can be mounted from a different device. Theprevious root (from initrd) is then moved to a directory andcan be subsequently unmounted. initrd is mainly designed to allow system startup to occur in two phases, where the kernel comes up with a minimum set of compiled-in drivers, and where additional modules are loaded from initrd.

When using initrd, the system typically boots as follows:

  1. The boot loader loads the kernel and the initial RAM disk
  2. The kernel converts initrd into a “normal” RAM disk and frees the memory used by initrd
  3. initrd is mounted read-write as root
  4. /linuxrc is executed (this can be any valid executable, including shell scripts; it is run with uid 0 and can do basically everything init can do)
  5. linuxrc mounts the “real” root file system
  6. linuxrc places the root file system at the root directory using the pivot_root system call
  7. The usual boot sequence (e.g. invocation of /sbin/init) is performed on the root file system
    8) The initrd file system is removed

 

Linux Operating System comes with Kill command to terminate a process. Linux provides a mechanism to interrupt a running program. Here I am explaining how a kill command will work, also some basic signals that we are using on a daily basis.

Externally generated Interrupts:

SIGKILL – if Control-C doesn’t work and nothing else seems to work then SIGKILL tells the Linux Kernel to take a sledge hammer to to the process and force it down. It is possible for running processes to ignore other signals and keep running, but SIGKILL can’t be ignored.
SIGINT – the results of a Control-C which normally cancels a running program
SIGHUP – traditionally line hangup , but is used now for telling a process to reinitialize itself
SIGTERM – termination request — normal default when someone issues a command like: kill 1298

SIGTERM is the default and safest way to kill a process. SIGHUP is less secure way of killing a process as SIGTERM. SIGKILL is the most unsafe way among the above three, to kill a process which terminates a process without saving.

Signal Actions
While there are several actions for the various signals on a Linux system.

Term – This action is used to signal that the process should terminate
Core – This action is used to signal that the process should core dump and then terminate
Common Signals
A list of a few common signals, the numeric value of that signal, the action that is associated with it.

  • SIGHUP – 1 – Term

The SIGHUP signal is commonly used to tell a process to shutdown and restart, this signal can be caught and ignored by a process.

  • SIGINT – 2 – Term

The SIGINT signal is commonly used when a user presses ctrl+c on the keyboard.

  • SIGQUIT – 3 – Core

The SIGQUIT signal is useful for stopping a process and telling it to create a core dump file. The core file can be useful for debugging applications but keep in mind your system needs to be setup to allow the creation of core files.

  • SIGKILL – 9 – Term

The SIGKILL signal cannot be ignored by a process and the termination is handled outside of the process itself. This signal is useful for when an application has stopped responding or will not terminate after being given the SIGTERM command. This signal should stop more processes however there are exceptions, such as zombie processes.

  • SIGSEGV – 11 – Core

The SIGSEGV signal is generally sent to a process by the kernel when the process is misbehaving, it is used when there is an “Invalid memory reference” and you may commonly see a message such as segmentation fault in log files or via strace. You can also technically call this signal with kill as well; however it is mainly useful for creating core dump files, which can also be performed by using the SIGQUIT signal.

  • SIGTERM – 15 – Term

The SIGTERM signal is the default signal sent when invoking the kill command. This tells the process to shutdown and is generally accepted as the signal to use when shutting down cleanly. Technically this signal can be ignored, however that is considered a bad practice and is generally avoided.

How Kill work with signals.

Syntax:
kill -9 PID

When the kill command is run it is actually sending a singal to the process. By default the kill command will send a SIGTERM signal to the specified process.The SIGTERM signal tells the process that it should perform it’s shutdown procedure to terminate the process cleanly by closing all log files, connections, etc.

In general it is a good idea for applications to close open file handles and external connections during shutdown, however sometimes these processes can either take a long time or due to other issues not happen at all. Leaving the process in a state where it is not correctly running but also not terminated.

When a process is in a limbo state it is reasonable to send the process the SIGKILL signal, which can be invoked by running the kill command with the -9 flag. Unlike SIGTERM the SIGKILL signal cannot be captured by the process and thus it cannot be ignored. The SIGKILL signal is handled outside of the process completely, and is used to stop the process immediately. The problem with using SIGKILL is that it does not allow an application to close its open files or database connections cleanly and over time could cause other issues, therefor it is generally better to reserve the SIGKILL signal as a last resort.

 

Reference :
http://man7.org/linux/man-pages/man7/signal.7.html
http://space.wccnet.edu/~chasselb/linux275/ClassNotes/process/sigbasics.htm
http://bencane.com/2014/04/01/understanding-the-kill-command-and-how-to-terminate-processes-in-linux/

Today I have faced yum error, after installing python3.3 instead of deafult python on the server. After installing the “yum” command is getting failed. So I have revert the python to default and enabled symlink for python3. Now the server has python3.3 and python2.6.6

root@ [/usr/bin]# yum install gcc
File “/usr/bin/yum”, line 30
except KeyboardInterrupt, e:
SyntaxError: invalid syntax

Enabled a symlink for python3

root@ [~]# ls -l /usr/bin/python
lrwxrwxrwx. 1 root root 9 Oct 24 00:06 /usr/bin/python -> python2.6*
root@ [~]# ls -l /usr/bin/python3
lrwxrwxrwx. 1 root root 32 Oct 24 00:15 /usr/bin/python3 -> /usr/local/python3/bin/python3.3*

root@ [~]# which python
/usr/bin/python
root@ [~]# which python3
/usr/bin/python3

root@ [~]# python -V
Python 2.6.6
root@ [~]# python3 -V
Python 3.3.2
root@ [~]# python3
Python 3.3.2 (default, Oct 16 2014, 06:30:05)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

root@ [~]# python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.

How a  Mail Server works

Introduction

images

 

This article is an introduction to Mail Server and it’s related protocols. Here I am explaining some basic ideas how a e-mail works and background information you need.

Electronic mail (email) is the term given to an electronic message, usually a form of simple text message, that a user types at a computer system and is transmitted over some form of computer network to another user, who can read it.

Email has become one of the driving forces behind connecting businesses to the Internet. It offers fast, economical transfer of messages anywhere in the world. Email messages doesn’t require paper or resources other than storage space thus we can support Green Environment policy.

Terminology

I think you heard about MUA, MTA and MDA while doing Linux training sessions. Do you know what is the exact role for these programs in a mail server? Here we are explaining in detail.

Mail User Agent (MUA) is mail client like Thunderbird,outlook, that allows a user to compose send and receive email.

Mail Transfer Agent (MTA) The MTA basically acts as a “mail router”. Which is responsible for sending the email to the recipient MTA. An MTA transfers mails via SMTP protocol. Later we explain in detail.

Mail Delivery Agent The recipient’s MTA receives the email and passes it on to a MDA. An MDA manages the user’s mailbox and handles the mails to deliver the MUA ie, (mail client) using either the POP3 or IMAP protocol. In other words, the MDA reads the header and sent back to a remote machine for email delivery.

How a Mail Server Works

We all have an email address. Do you ever think how an email works while we sending an e mail to some other recipient address. When we are clicking on the email send button, within minutes the email will deliver to the destination. But at first we have no idea, what are process take place behind the scene and to make sure that the email reaches it’s final destination. Here we are giving an introduction on various back end process of a Mail Server.

1 .When you compose an email with the help of Email Client, the email client will connect to your SMTP server the configuration that we have provided, while configuring your email client.

  1. For an example here the SMTP server is (mail.example.com). Once the email client has connected to mail.example.com SMTP server, it will forward the message that you have just composed to the server.
  2. An SMTP server is also called as MTA. Why we called as an MTA, because it’s works as a messenger. For example it transfers the mails and messages to the required recipients SMTP server.
  3. Next step is the SMTP server will do a DNS query for getting the SMTP server of your email receiver. And the SMTP server will try to find the domain SMTP server detail and handed over the message for the particular user. For example when you are sending an email to user@example.com. In that case the SMTP server will try to find the responsible domain SMTP server detail, example.com and hand over the message for the user.
  4. Once your SMTP server finds the target receivers SMTP server, using an MX record lookup, it will forward the message to that server. Then the SMTP server will forward the message to POP3 or IMAP server responsible for the domain.

Mail server functionality can be divided into two processes:

Sending and Receiving emails

Sending email : Simple Mail Transfer Protocol (SMTP)
Receiving email : Post Office Protocol (POP3) / Internet Message Access Protocol (IMAP)

What is an SMTP protocol and why we are using this protocol in mail servers?

SMTP (Simple Mail Transfer Protocol) is a TCP/IP protocol used in sending and receiving e-mail. In simply the purpose of SMTP is to set up communication rules between servers, like (postman). When we are sending a mail to some other recipient address, the connection happens via SMTP protocol.

Comparison between IMAP and POP3 protocol

IMAP and POP3 are two different protocols

The main difference is that IMAP(Internet Messaged Access Protocol) always syncs with mail server. So whenever you made any changes on your mail client, the result will instantly appears on your inbox.

The biggest advantage of using IMAP is you can access your mail from multiple mail clients like Thunderbird and each mail client will shows real-time update. Suppose mail server is connected with two different mail clients on different computers. If one of the user deletes a message from mail client, the changes will immediately update on both mail server and client.

This why IMAP is more suitable and if you’re going to access your emails from different locations or if your mails are managed by multiple users.

In POP3 (Post Office Protocol), your mail client (Thunderbird) and mail server will not synced. This means the mails are downloaded in your computer and the changes won’t reflect on the server.

In POP3 multiple mail clients access were not supported. Here you can download emails from mail server to your computer using POP3. After downloading, the original mails were removed from the server. Hence you can’t access the mails from another computer.

Mail Service protocols and ports

Simple Mail Transfer Protocol (SMTP)

Port 25 – SMTP non-encrypted port
Port 465 – SMTP secure port

Post Office Protocol version 3 (POP3)

Port 110 – POP3 non-encrypted port
Port 995 – POP3 secure port

Internet Message Access Protocol (IMAP)

Port 143 – IMAP non-encrypted port
Port 993 – IMAP secure port

MX Record & priorities

Mail exchange is just another name for the machine whose primary function is receiving and sending email. An MX record is set to point a canonical hostname, like example.com or mail.example.com. You can find your domain mail server detail either from dig or online tools, like intodns.com and mxtoolbox.com

The MX record uses preference values to specify the routing order –low value = high priority. The MX priority determines the order (which mail server) that your mail will be attempted to be delivered. The mail server with the lowest MX priority will first be attempted to deliver.

google.com. 600 IN MX 40 alt3.aspmx.l.google.com.
google.com. 600 IN MX 30 alt2.aspmx.l.google.com.
google.com. 600 IN MX 50 alt4.aspmx.l.google.com.
google.com. 600 IN MX 10 aspmx.l.google.com.
google.com. 600 IN MX 20 alt1.aspmx.l.google.com.
So, if you have five MX records with levels 10, 20, 30, 40 ,50 the following would occur. Mail would always be first tried to be delivered to the MX record with MX priority of 10. If that mail server is down then the mail will try to be delivered to the mail server at 20. If the mail server at priority 20 is down then the mail will be attempted to be delivered at the mail server at priority 30. If you have multiple MX records with the same MX priority, then it will setup a round robin configuration for your email.

Types of Mail Service

Dovecot
Courier

In cPanel servers we are using dovecot and courier mail services. This is the service which delivers the email to your inbox while exim is the one which sends the mails to the servers.

Dovecot

Dovecot uses less memory,better performance and is more configurable. Dovecot is a POP3 and IMAP mail server that can work with standard mbox and maildir formats. Dovecot is much faster than Courier and the advantage is its intelligent use of configuration files.

Files

• Program : /usr/sbin/dovecot
• Init Script : /etc/init.d/dovecot
• Config : /etc/dovecot.conf

Courier

Courier is extremely reliable, but it needs a larger memory. Why it takes heavy memory usage, because when ever you open web mail client it has to reload the entire inbox. Which makes it a bit slower to load compared to dovecot, when we are dealing with large mailboxes.

Files

• Program : /usr/lib/courier-imap/libexec/couriertcpd
• Init Script : /etc/init.d/courier
• Config : /usr/local/libexec/courier-imap

If you are running a shared server, dovecot need to deliver better performance while you have a dedicated server for one or two websites, courier can be more efficient. The default choice for cPanel is the dovecot and if you need to change you need to navigate to WHM Main >> Service Configuration >> Mailserver Selection

OR

You can change it from back-end command line using cPanel script and editing config file.

/scripts/setupmailserver dovecot

Then set /var/cpanel/cpanel.config

maildir=1
mailserver=dovecot
Exim Configuration files

Exim is an open source mail transfer agent (MTA), which is a program responsible for Receiving, Routing, Delivering e-mail messages

Exim commands

exim -bp: shows messages in queue
exim -bpc : shows the no.of messages in queue
exiwhat : Finding out what Exim processes are doing
exim -qff : Attempt to flush frozen messages
exim -bp | exiqsumm : Print a summary of messages in the queue

exiqgrep usage

exiqgrep -f [luser]@domain : Search the queue for messages from a specific sender
exiqgrep -r [luser]@domain : Search the queue for messages for a specific recipient/domain
exiqgrep -i : To Print just the message-id of the entire queue
exim -Mrm: <message-id> : Remove a message from the queue
exim -M <message-id>: Deliver a specific message
exiqgrep -z -i | xargs exim -Mrm: Remove all frozen messages

Files

/etc/exim.conf – mail server configuration file
/etc/localdomains – exim related file. All the domains using the same server’s MX, should be listed here to be able to send/receive emails.

/etc/valiases/domainname – catchall and forwarders are set here.
/etc/vfilters/domainname – email filters are set here.
/var/spool/exim/input – Mail queue.
/var/spool/exim/msglog – email message logs.
/var/cpanel/horde – version file, backups and logs stored here
/var/cpanel/roundcube – version file, backups and logs stored here
/var/cpanel/squirrelmail – version file stored here.

You can also edit and modify exim configuration from WHM

Home >> Service Configuration >> Exim Configuration Manager >> Advanced Editor

How to change exim Mail server IP address

When ever our main server is blacklisted in RBL sites. This is only a temporary work around to the blacklist problem and you have to make sure that you have identify the spammer and resolved the issue. As a temporary solution we can change the mail server IP address, this resolve the email bounce back issue. If you have any email script under the domains, then you need to change the outgoing mail IP address on that script. In some cases PHP mailer script will give you bounce back message after changing the mail server IP address.

Changing the IP Globally

You need edit the following file /etc/mailips

*: xxx.xxx.xxx.xxx

Then add the IP and it’s matching PTR to /etc/mail_reverse_dns:

xxx.xxx.xxx.xxx hostname.tld

This will tell Exim to use that IP for any sender on the server.

Spamassin

Apache SpamAssassin is an email utility that examines incoming email and tests for spam characteristics. SpamAssassin is designed to identify and mark e-mails that score beyond your threshold value. SpamAssassin has 10 different levels of settings to catch spam. By default the spam score will be 5. You can enable the SpamAssassin from the domain cPanel itself.

How spam score works

If you set lower the score, more email will be caught as spam. For example, you have enabled spam score as “1” that means only one hit needs to be flagged against the email to be considered as spam. If you set the spam score higher, more hits will be required on an email for it to be labeled as spam. So if we set lower spam score, more emails should be flagged as spam.

0 means everything incoming will be marked as spam.
5 is the default setting (and works well for typical users).
10 means that any message with a score of 10 or less will not be marked as spam.
RBL

What is an RBL ?

A DNS-based Blackhole List or Real-time Blackhole List is a list of IP addresses which are most often used to publish the addresses of computers or networks linked to spamming, most mail server software can be configured to reject or flag messages which have been sent from a site listed on one or more such lists.

Eg: spamhaus,Spamcop etc

How to check whether an IP is blacklisted ?

You can check whether the server mail IP address is blacklisted in any RBL’s using the below link

Eg: http://mxtoolbox.com/blacklists.aspx

How to delist the IP address

For delisting you need to contact the blacklist provider from there you can check whether the IP address was listed on their database. eg, http://www.spamhaus.org/lookup/ .It will take 48 hours to complete the delist process.

How we can prevent spamming activities and what are the methods ?

We all know about spam mails right? We are getting so many spam mail in our personal mails, but in Gmail,yahoo and other service providers have their own techniques to prevent spam mails. Here I am sharing a brief description about two different technologies in cPanel SPF and DKIM.

Whenever you create a domain on CPanel server using WHM, it won’t add domainkeys and SPF records for particular domains. We need to add it manually.

Sender Policy Framework (SPF)

SPF helps prevent spammers, SPF can also reduce the amount of bounce messages that you receive. SPF uses DNS records that specify the mail servers and IP addresses that are authorized to send e-mail messages from a domain.

To Enable SPF Records for a domain

Log into cPanel
Click Email Authentication
Click ENABLE beside the SPF section.
Click the Go Back link
Verify that the Status now says that it is Active and Enabled

You can add from command prompt too.

/usr/local/cpanel/bin/domain_keys_installer “domain username”

DomainKeys Identified Mail (DKIM)

You can use DKIM to verify an incoming e-mail message is actually from the stated sender, and that the message has not been altered during transit. When DKIM is enabled, the sender digitally signs a message using a private key. The recipient uses DNS to retrieve the sender’s public key and verify the message’s signature. If the signature is invalid, then the message is assumed to be forged and therefore spam.

To Enable Domain Keys

Log into cPanel
Click Email Authentication
Click ENABLE beside the Domain Keys section
Click the Go Back link
Verify that the Status now says that it is Active and Enabled

You can add from command prompt too.

/usr/local/cpanel/bin/domain_keys_installer “domain user name”

How to check e-mail log using exigrep

Here I am explaining how to trace and identify the mail log symbols. One of the best tools you can use when tracking down e-mail problems is mail logs.

First, it is a good idea to get to know the following symbols.

<=
=>
->
*>
**

At least one of these symbols will be on nearly every line of exim’s mainlog. Other abbreviations in the log will change their meaning based on which of these symbols is on that line. These abbreviations consist of one or two letters.

2014-10-07 03:41:47 [4578] 1XbPOr-0001Bq-OC H=ccm181.constantcontact.com [208.75.123.181]:58477 I=[72.34.47.10]:25 Warning: “SpamAssassin as ahmedkha detected message as NOT spam (-4.6)”

2014-10-07 03:41:47 [4578] 1XbPOr-0001Bq-OC <= A2tcbHRYaQHK0cJzWtAZ5dg==_1110613794467_pL+XcLVEEeOkRNSuUnVJUA==@in.constantcontact.com H=ccm181.constantcontact.com [208.75.123.181]:58477 I=[72.34.47.10]:25 P=esmtp S=37537 M8S=0 id=1118748363623.1110613794467.22409.0.210333JL.1002@scheduler.constantcontact.com T=”Ne ratez pas le RV incontournable du business de la television en\n Afrique !” from <A2tcbHRYaQHK0cJzWtAZ5dg==_1110613794467_pL+XcLVEEeOkRNSuUnVJUA==@in.constantcontact.com> for ahmed@ahmedkhaled.com

Beginning of the Line

Each line starts with the date and time. Immediately following is exim’s internal message ID:
First Line

The next item on the first line starts with “H=”. This specifies the host name of the server that the mail originates from. Next (in square brackets) is the IP address of that server, followed by the port number. “I=” looks similar, but is describing your mail server. A warning follows: Spamassassin believes that this message is not spam.

Second Line

The second line almost starts the same (with H and I), but before that the “<=” symbol is used. This indicates that this line is describing the arrival of the message on your mail server. Immediately after this symbol is the e-mail address that this mail is being sent from.

After H and I, the “P” abbreviation designates the protocol being used (here esmtps). Note that this only means this because it is on a “<=” line; if this were a “=>” line it would indicate the return path of the message.

The X tells us the particular cipher suite that is used; this is usually not something you would be looking for when troubleshooting a mail issue. CV refers to certification verification status, and S is the size of the message.

Next, “id” refers to the message id that was created by the sending server and sent as one of the mail headers. T (for topic) is the subject of the e-mail. The line ends with “for user@example.com”, letting us know just who the message is for.

Steps to troubleshooting spamming

https://linuxtechme.wordpress.com/2012/04/23/spamming-in-exim/

Mail Server

SPF Record Syntax

Posted: 4p in cPanel

SPF Record Syntax

The Sender Policy Framework (SPF) is an open standard specifying a technical method to prevent sender address forgery.

“+” Pass
“-“ Fail
“~” SoftFail
“?” Neutral

 

“v=spf1 -all”If a mechanism results in a hit, its qualifier value is used. The default qualifier is “+“, i.e. “Pass”. For example:

"v=spf1 a -all"

"v=spf1 a mx -all"

"v=spf1 +a +mx -all"

Mechanisms are evaluated in order. If no mechanism or modifier matches, the default result is “Neutral”.

If a domain has no SPF record at all, the result is “None”. If a domain has a temporary error during DNS processing, you get the result “TempError” (called “error” in earlier drafts). If some kind of syntax or evaluation error occurs (eg. the domain specifies an unrecognized mechanism) the result is “PermError” (formerly “unknown”).

Evaluation of an SPF record can return any of these results:

Result Explanation Intended action
Pass The SPF record designates the host to be allowed to send accept
Fail The SPF record has designated the host as NOT being allowed to send reject
SoftFail The SPF record has designated the host as NOT being allowed to send but is in transition accept but mark
Neutral The SPF record specifies explicitly that nothing can be said about validity accept
None The domain does not have an SPF record or the SPF record does not evaluate to a result accept
PermError A permanent error has occured (eg. badly formatted SPF record) unspecified
TempError A transient error has occured accept or reject

 

The “ip4” mechanism

ip4:<ip4-network>/<prefix-length>

The argument to the “ip4:” mechanism is an IPv4 network range. If no prefix-length is given, /32 is assumed (singling out an individual host address).

Examples:

“v=spf1 ip4:192.168.0.1/16 -all”

Allow any IP address between 192.168.0.1 and networks

 

cPanel logs

Access logs and user actions                                     /usr/local/cpanel/logs/access_log
Account transfers and misc. logs                             /var/cpanel/logs
Auditing log (account creations, deletions, etc)  /var/cpanel/accounting.log
Backup logs                                                               /usr/local/cpanel/logs/cpbackup
Brute force protection (cphulkd) log /usr/local/cpanel/logs/cphulkd.log
Cpanel dnsadmin dns clustering daemon /usr/local/cpanel/logs/dnsadmin_log
Cpanel taskqueue processing daemon /usr/local/cpanel/logs/queueprocd.log
DBmapping /usr/local/cpanel/logs/setupdbmap_log
EasyApache build logs /usr/local/cpanel/logs/easy/apache/
Error log /usr/local/cpanel/logs/error_log
Installation log /var/log/cpanel
License updates and errors /usr/local/cpanel/logs/license_log
Locale database modifications /usr/local/cpanel/logs/build_locale_database_log
Login errors (CPSRVD) /usr/local/cpanel/logs/login_log
Horde /var/cpanel/horde/log/
RoundCube /var/cpanel/roundcube/log/
SquirrelMail /var/cpanel/squirrelmail/
Panic log /usr/local/cpanel/logs/panic_log
Per account bandwidth history (Cached) /var/cpanel/bandwidth.cache/{USERNAME}
Per account bandwidth history (Human Readable) /var/cpanel/bandwidth/{USERNAME}
Service status logs /var/log/chkservd.log
Tailwatch driver tailwatchd log /usr/local/cpanel/logs/tailwatch_log
Update analysis reporting /usr/local/cpanel/logs/updated_analysis/{TIMESTAMP}.log
Update (UPCP) log /var/cpanel/updatelogs/updated.{TIMESTAMP}.log
WebDisk (CPDAVD) /usr/local/cpanel/logs/cpdavd_error_log
Website statistics log /usr/local/cpanel/logs/stats_log

cPanel access log

Access logs and user actions /usr/local/cpanel/logs/access_log

cPanel apache log

Apache restarts done through cPanel and WHM /usr/local/cpanel/logs/safeapcherestart_log
Domain access logs /usr/local/apache/domlogs/{DOMAIN}
Processing of log splitting /usr/local/cpanel/logs/splitlogs_log
suPHP audit log /usr/local/apache/logs/suphp_log
Web server and CGI application error log /usr/local/apache/logs/error_log
cPanel email log

Delivery and receipt log                                                               /var/log/exim_mainlog
Incoming mail queue                                                                    /var/spool/exim/input/
Log of messages rejected based on ACLS or other policies  /var/log/exim_rejectlog
Unexpected/Fatal error log                                                          /var/log/exim_paniclog
IMAP, POP login attempts, transactions, fatal errors and spam scoring /var/log/maillog /var/log/messages
Mailman                                                                                         /usr/local/cpanel/3rdparty/mailmain/logs
MySQL log

MySQL error log                                                        /var/lib/mysql/{SERVER_NAME}.err
MySQL slow query log (if enabled in my.cnf)     /var/log/slowqueries

How to run a MySQL benchmarking using sysbench

Installing sysbench

#yum install sysbench

The next step is to use the prepare statement with sysbench to generate a table in the specified database which will be used when performing tests.

root@# sysbench –test=oltp –oltp-table-size=250000 –mysql-db=database name –mysql-user=db user –mysql-password=db password –mysql-socket=/tmp/mysql.sock –db-driver=mysql –oltp-auto-inc=off prepare

sysbench 0.4.12: multi-threaded system evaluation benchmark

Creating table ‘sbtest’…
Creating 250000 records in table ‘sbtest’…

The above command has created a table called sbtest with 250000 rows of data which will be used for testing. The below commands show the the created table.

root@# sysbench –test=oltp –oltp-table-size=10000 –oltp-test-mode=complex –oltp-read-only=on –num-threads=1000 –max-time=300 –max-requests=5000 –mysql-db=database name –mysql-user=db user –mysql-password=db password –mysql-socket=/tmp/mysql.sock –db-driver=mysql run

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1000

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using “BEGIN” for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 5000
Threads started!
Done.

OLTP test statistics:
queries performed:
read: 70056
write: 0
other: 10008
total: 80064
transactions: 5004 (1128.50 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 70056 (15799.05 per sec.)
other operations: 10008 (2257.01 per sec.)

Test execution summary:
total time: 4.4342s
total number of events: 5004
total time taken by event execution: 4294.8422
per-request statistics:
min: 34.05ms
avg: 858.28ms
max: 2907.31ms
approx. 95 percentile: 2281.78ms

Threads fairness:
events (avg/stddev): 5.0040/0.73
execution time (avg/stddev): 4.2948/0.07
=-=-=-=

Finally, you can drop the database that have created while doing test.