Tagged with " programming"
Apr 17, 2013 - Development    Comments Off

PHP Email Bounce Script

A lot of people have asked for this. This is the email bounce script I use. You will have to pipe all emails to this file.

<?
//opens the stream
$fd = fopen("php://stdin", "r");

$content = "";

//read the lines
while (!feof($fd))
{
    $content .= fread($fd, 1024);
}
fclose($fd);

$lines = explode("\n", $content);

$email = "";

//SPAM report from feedback loop
for($i=0;$i<count($lines);$i++)
{
    if(preg_match("/^Original-Rcpt-To:(.*)/", $lines[$i], $matches))
    {
        $email = trim($matches[1]);
        $mode = "spam";
        $type = 2;
        continue;
    }
}

//bounced back message from receiving server
if($email == "")
{
    for($i=0;$i<count($lines);$i++)
    {
        if(preg_match("/^Delivered-To:(.*)/", $lines[$i], $matches))
        {
            $mode = "bounce";
            $type = 1;
            $email = substr(trim($matches[1]),2);
            continue;
        }
    }
}

//failed message, MTA emailed a failed message
if($email == "")
{
    for($i=0;$i<count($lines);$i++)
    {
        if(preg_match("/^Remote host said:(.*)/", $lines[$i], $matches))
        {
            for($j=0;$j<count($lines);$j++)
            {
                if(preg_match("/^Return-Path:(.*)/", $lines[$j], $matches))
                {
                    if(strstr($matches[1], "=") !== false)
                    {
                        $mode = "bounce";
                        $type = 1;
                        $email = trim($matches[1]);
                        continue 2;
                    }
                }
            }
            continue;
        }
     }
}

//return-path format is user=website.com@mydomain.com
//convert this into the actually email, user@website.com
if($mode == "bounce")
{
    $email = substr($email,0,strpos($email,"@"));
    $email = str_replace("=", "@", $email);
}

if($email != "")
{
    if(strchr($email, "@") !== false)
    {
        $domain = substr($email,strpos($email,"@")+1);
        $sql = "SELECT id
                FROM donotemail
                WHERE LOWER(email) = LOWER('".fix($email)."')";
        $result = query($sql);
        if($result)
        {
            if(numrows($result) == 0)
            {
                $sql = "INSERT INTO donotemail
                        (email, type, thedate)
                        VALUES('".fix($email)."', '$type', CURDATE())";
                query($sql);
            }
        }
    }
}
?>
Mar 12, 2013 - Development    Comments Off

PHP date_diff days negative zero issue

I ran into an odd problem today that I couldn’t find a solution to when searching Google. I ended up figuring it out and thought I should document it here for future searches.

PHP has a date_diff function from v5.3+. I was using this function to calculate the difference in DAYS between two dates. I found that the results I received did not match what I expected. PHP says the difference between Yesterday and Today is -1 (negative one) days. The difference between Today and Today is -0 (negative zero) days. The difference between Tomorrow and Today is 0 (zero) days. And the difference between Today and the Day After Tomorrow is 1 (one) day, etc. Say whaaaaat?

Duplicate this:

$day1 = new DateTime();
$day2 = new DateTime("yesterday");
$interval = $day1->diff($day2);
$interval = date_diff($day1, $day2);
$days = $interval->format('%r%a');
//%r = show negative sign or nothing
//%a = number of total days

echo "$days <br/>";

$day2 = new DateTime("today");
$interval = $day1->diff($day2);
$interval = date_diff($day1, $day2);
$days = $interval->format('%r%a');

echo "$days <br/>";

$day2 = new DateTime("tomorrow");
$interval = $day1->diff($day2);
$interval = date_diff($day1, $day2);
$days = $interval->format('%r%a');
echo "$days <br/>";

In order to get a difference value that I expected I had to write a statement to convert these values. I also ran into some odd comparison issues and had to resort to using === to compare the days. Even though I compared string to string I kept getting false positives. I threw together a quick conversion if/then/else that created the right type of “days” value for what I was expecting:

if($days === "-0")
{
//today
$days = 0;
}
elseif($days === "0")
{
//tomorrow
$days = 1;
}
elseif($days > 0)
{
$days = $days + 1;
}

PHP handled the other days in the past as expected, for whatever reason. The day before yesterday is -2 without any modifications.

Mar 1, 2013 - Development    Comments Off

Domain Name Availability / WHOIS API

On Naming Force we need the ability to check the availability of a suggested domain names. Previously we had used the eNom API to do this but received many complaints that the data was inaccurate. Also, we ran into an issues where eNom was blocking our API requests if we submitted “too many” in a short time period. They would flag our account for 30 minutes to an hour, during which we could perform no API calls. I contacted eNom support and they said there was no way to get around this and that they also would not tell us what the threshold was. Horrible idea for an API service, eNom. I searched for a replacement and came upon WhoisAPI. We have been using WhoisAPI for about a month now and it seems to be much better than the eNom API for checking availability of domains. The WhoisAPI is more expensive, but it’s tailored to exactly what we want to accomplish. I recommend WhoisAPI for anyone trying to find an API for checking availability of domain names.

Jun 27, 2012 - Business    Comments Off

Facebook API: use publish_actions instead of publish_stream

Don’t use publish_stream, use publish_actions!

I was doing some research for Fido Finder’s Neighborhood Watch feature and I came upon the actual documentation for the different Facebook Graph API stream publishing permissions. I was shocked to find that every single HOW TO article I’ve read online is telling people to request the unnecessary publish_stream permission which actually requires the user to agree to additional permissions on a 2nd authorization screen (sound familiar?).

Publish_stream actually includes the publish_actions permissions and adds a few more permissions such as posting to a friend’s feed, posting questions, creating notes, and posting content to events or groups. If you do not need these 4 extra permissions then you DON’T NEED publish_stream. And, this means that users don’t have to agree to that scary “post to your feed” permission even though it’s actually included.

Every app I’ve ever authorized had a 2nd screen like:


The user can SKIP this permission!

If you use publish_actions you simply get the 1st screen:

Jun 26, 2012 - Business, Development    Comments Off

PHP Email Sending Guide

I have run into a lot of issues over the years getting my website-generated emails delivered to inboxes. Unfairly marked as SPAM, accidentally blocked by ISP’s, and occasionally lost in space. This guide is a collection of what I’ve learned about sending emails from PHP. I’m going to keep this entry updated as I learn more tips.

Use PHPMailer

I use PHPMailer which allows me to send emails by signing into the SMTP account/server (if you tell it to). I also have more control over headers, etc, with PHPMailer. When I switched to using PHPMailer/SMTP I noticed my messages not being marked as SPAM with certain email providers.

Send From Dedicated IP

Make sure your SMTP server does not share an IP address with others. If you host in a shared environment you’re out of luck. Even some dedicated server environments share an SMTP server (my old environment did, and I left). If your neighbor sends spam from the same server that you use for outgoing mail, your mail will likely be marked as SPAM, or worse, not delivered at all. Sometimes this lasts 24 hours, sometimes longer. Few businesses can risk not having their emails delivered.

Add DNS Entries

Email servers will check your DNS entries to make sure your outgoing IP is allowed to send mail for the domain.

  • Reverse DNS (rDNS / PTR) record -  Standard DNS resolves a domain name to an IP address. Reverse DNS resolves an IP address to a domain name. It says, OK 98.129.58.195, what’s your domain name? This is used to help determine the origin of the message and whether it matches the sending domain/address.
  • SPF (Sender Policy Framework) record – The idea behind an SPF record is for your domain server to be able to be queried and asked for a list of email servers that are allowed to send email for your domain. Even if you’re sending emails from the same server IP address, it helps to project to the email servers out there that your emails are allowed to be created from this IP address for your domain name.
  • DomainKeys and/or DKIM – Don’t be confused (by the Internet), these are two different, but similar, technologies DomainKeys and DKIM work in a public-key scenario, so a key is sent in the email that can be verified by querying the DNS server. Google it, or contact your webhost to find out how to set this up. It will be a combination of server settings and DNS entries. I was able to set up DomainKeys via Plesk but have not been able to figure out setting up DKIM.

HELO

Make sure your HELO, in the email header, does not show a different domain than the sending server. In my case, and maybe this is always true, when I finally got the HELO setup correctly the HELO message disappeared from the header, as it was no longer was needed. In my configuration I had to setup the HELO to match the sending server name which is not the domain of the email address. I set this in the PHPMailer config settings for HELO.

Odd SPAM Filters

  • Even if you’re sending mail from a do-not-reply address, you need to have a return address in the email headers. Many email hosts will not deliver mail that does not have a reply to address.
  • Do not obfuscate your HTML. Make sure it isn’t obvious that you created your HTML in code (one long line, etc).
  • Be careful with images. In one instance the occurrence of a simple logo at the top of outgoing emails led GMail to send the emails to the SPAM folder. Avoid them unless you need them.
  • Don’t use words like “100% guaranteed”, “as seen”, or an all caps subject.
  • Look through SpamAssassin header rule codes to learn more.

Handle Bounced AND Rejected Emails

  • Bounced – Some mail will be bounced back to you at a later time/date. Set up VERP for outgoing emails. This will mean adding a header of “return-path” to outgoing emails that includes the receiving email address in a specially formatted string. This string appears to be an email address at your domain but in reality routes to your catch-all email box. Then, you have your server set to pipe all mail to the catch-all box to a PHP script. The special string looks like user=domain.com@yourwebsite.com. You can parse this string and discover what address you emailed that bounced back. In PHPMailer I had to modify the code to allow for a new property called “returnpath” and I modified the code where PHPMailer sets the returnpath header value. Email me for more info.
  • Rejected – Some mail will fail to send immediately, but PHPMailer won’t know this (for whatever reason). Since your MTA will email you a rejection notice, have all of your email sent to your do-not-reply address piped to the same script. When your MTA sends you a failed delivery notice you will want to process this message and add the email address to your do-not-email database table.
  • Here is a link to my php email bounce script. I found this online and modified it over the years.

Sign up for Feedback Loops

The major mail providers and ISPs offer a feedback loop. Feedback loops allow the mail providers to alert you when a user reports your messages as SPAM. You can then act accordingly (stop emailing them?). Here is a list of some of the available feedback loops.

Jun 10, 2012 - Business, Development    Comments Off

Handling Bounced Emails in PHP

I’ve touched on this subject before, but I’m going to go more in-depth as I recently learned that I wasn’t fully handling bounced emails. The problem I run into is that there is no manual for properly running a website. You can read all you want about HTML, CSS, PHP, MySQL, but you’ll be hard-pressed to find good books and articles on how to run a website successfully. Do’s and dont’s. How to avoid problems. You have to learn piece-by-piece as you make mistakes.

Why Bounces are Important

As I’ve talked about before, it’s important to handle bounced emails because ISP’s and email providers use this information against you. If you send emails to invalid, expired, or spamtrap email addresses, you will be punished. Your emails will not be delivered with the regularity that you expect.

Two Types of Bounced Emails

I just learned, after more than 2 years of thinking I was properly handling bounced emails sent from my websites, that I was not handling the majority of bounced emails. This means that my websites are emailing mistyped, expired, and fake email addresses daily, without my knowledge. Technically, there are bounced emails and there are rejected emails.

#1 – Bounced Emails

I thought I was so smart when I first set up bounced email handing. I don’t recall where I read about how to do this in PHP, but what I read was that I first of all needed to add a return-path header to outgoing emails. This return-path header will contain a way for your site to discern who the intended recipient was. On my sites I formatted the return-path header as such, “user=domain.com@mywebsite.com”. I then have to set all emails coming to my website, that are addressed to a nonexistent address, to be sent to a new email inbox. Then, this inbox has to be “piped” to a specific PHP page. The PHP page reads all emails it receives, checking for this return-path header. It parses the header, converts it to properly formatted email address, and adds the email to a do-no-email table in the database. This is great, and needed, but bounced emails going to return-path addresses make up only a part of invalid or bounced email attempts. I guess, I don’t know for sure, but I assume that this type of bounce happens when an address is recently deleted, only. Addresses that simply never existed, or have been gone for some time, appear to not bounce in this way. Thanks SMTP geeks.

#2 – Rejected Emails

It would be too easy for the above to be the end-all way to handle bounces. In doing some research for why some ISP’s hate emails from FidoFinder.com, I was browsing through my SMTP email log for a specific day. I saw some 550 errors where I was attempting to email invalid email addresses. I wanted to verify that these addresses had been added to the do-not-email database table after this bounce, and none of them had. I then wrote a test page to email these same addresses. I added a line to my bounce PHP code to email me a copy of any email it received. I never got a bounce emails. After a short discussion with Rackspace in the support ticket I had open for getting a copy of that SMTP log file, something the tech said revealed to me that many times the server rejects the attempted email, and that this does not produce an email to the return-path address. Instead, the MTA, in this case Qmail, sends the sender a message telling him about the bounce. Everyone has received a similar email before. The problem was I had my outgoing website address set to simply delete all incoming mail, as it was a do-not-reply address that people were not supposed to send emails to. In order to read these reject messages I had to remove the setting to pipe all emails to NULL and now pipe these emails to my bounce script. I added some code to look for “Remote host said”, a common text in email rejection notices, and consider all emails with this line to be reject bounces. I then search for the return-path header, which is included as the part of the original email, and add this address to the do-not-email database table. Now we’re adding rejected addresses to the do-not-email table.

Conclusion

In the first day I had 10-15 rejected emails where I normally have 1 bounced email. This means I was barely just scratching the surface of handling bounced emails previously. Possibly the repeated attempt to send emails to nonexistent emails has been affecting Fido Finder’s delivery rate for years. That was the idea when I set up the first bounce rules. So hopefully cutting down the emailing of these rejected addresses will have a positive effect.

Mar 7, 2012 - Development    Comments Off

More Email Delivery Tips

Recently one of my websites sent me an email that showed up in my Spam folder. I viewed the headers to see the SpamAssassin flags and was a little shocked. Two things were a complete surprise.

First, I learned that there was a flag for obfuscated HTML. You know, HTML with no carriage returns. When I create HTML emails in PHP I absolutely don’t add random carriage returns – I’m never ever going to actually look at this HTML!!! Big mistake! Apparently Spam filters think obfuscation is spammy. Blah. So I went into all of my PHP HTML email code and added \n to spots where I would normally have a carriage return in a normal HTML file. That removed this flag.

The next flag was for having too many bytes of image compared to the bytes of text. Are you kidding me? I have a small 300×60 6.5K logo at the top of emails. I’m being punished for using my logo and not writing super-long emails to offset this? I removed the logo, added an H1 tag with the domain name, and the flag went away. Side note, emails were not going to my test GMail inbox and suddenly when I removed the logo they did. I don’t think they use SpamAssassin, but it’s worth noting that their own Spam detection software did not like my small image in the emails.

Fixing both of these “issues” allowed SpamAssassin to deliver the exact same email without it being marked as Spam. I never knew that a small  image and lazy HTML would lead to emails not being delivered to inboxes. I thought I had all this figured out by now. So wrong.

If you want to track HTML email views in Google Analytics, read my blog entry here. For a much more detailed list of things to do to increase email delivery, read this older blog entry.

Mar 7, 2012 - Development    Comments Off

Tracking Email Views with Google Analytics

If you send HTML emails it’s possible to track their views through Google Analytics (GA). Although most email applications do not run Javascript, you can actually record GA data on the server side. This is an example of implementing view tracking for emails in PHP.

Google offers GA code for mobile apps as well as mobile browsers that do not run Javascript. You can record visits through an image whose source is a PHP script. You’ll be adding a tiny image to the end of your HTML emails that triggers the GA code. Google offers the script as an SDK. Download the SDK. The ga.php file doesn’t need to be modified for our example, but we did implement the rest of their code with some variations.

Our own goal was to simply track the increase and decrease in the viewing of emails sent from the website. We wanted to be able to see if changes to our email format affected email delivery. Since most email clients make the user enable images to be loaded in HTML emails, you won’t get true statistics on views, but over time you’ll definitely be able to notice increases and decreases in email viewing.

Below is our altered function that we call in order to create the URL needed to be placed in the image source. We wanted to only record the type of email that was viewed, not any URL/Referrer information like GA normally records. So we altered the function to remove the URL/Referrer code and to accept an $emailcampaign parameter which we populate with “email-something”.

$GA_ACCOUNT = “UA-XXXXX-1″;
$GA_PIXEL = “_ga.php”;

function googleAnalyticsGetImageUrl($emailcampaign)
{
global $GA_ACCOUNT, $GA_PIXEL;
$url = “”;
$url .= $GA_PIXEL . “?”;
$url .= “utmac=” . $GA_ACCOUNT;
$url .= “&utmn=” . rand(0, 0x7fffffff);

$path = $emailcampaign;

if(!empty($path))
{
$url .= “&utmp=” . urlencode($path);
}

$url .= “&guid=ON”;

return $url;
}

While we create the HTML string for the email we call googleAnalyticsGetImageUrl(“email-something”) and set the result as the “src” of an image tag placed at the end of the body of our HTML email. Then in GA we can search for Content like “email-” and see all of the views based on the type of email (campaign).

Dec 1, 2011 - Development    No Comments

Getting Facebook User Data with PHP

If you don’t already know, Facebook has opened up their data for some, regulated, 3rd party use. Facebook calls this their Open Graph API (application programming interface). Any developer who follows the specified steps can create applications that integrate with Facebook user data. Facebook does apply limitations on how developers can use the data in order to protect Facebook users. But, for the most part, if a Facebook user agrees, you can gain access to just about all of their profile data, and even post to their wall, etc.

Getting Started with Facebook User Data in PHP

Before you can start developing with Facebook you’ll need to register an application (yet to be developed) with Facebook. Start on the Facebook Developers page and then go to the Apps tab. I’m not going to go into detail about this because it’s pretty straight forward and already written about everywhere on the web. To make getting to the data easier there is a PHP SDK for Facebook Open Graph. We will be using this library for all of our Facebook queries.

Permissions

Before you can get to any data a user is going to first have to grant your application access to specific parts of their data. Facebook has different data and actions sectioned into permission groups. Below is an example of requesting permissions to get a user’s email address, bio data (name, etc), and to be able to read their wall.

$config = array(
‘appId’ => FBAPPID,
‘secret’ => FBAPPSECRET,
);

$facebook = new Facebook($config);
$fbuserid = $facebook->getUser();

$params = array(
“scope” => “read_stream,email,user_about_me”,
“redirect_uri” => “http://www.wescutshall.com/”
);

$loginurl = $facebook->getLoginUrl($params);

This will result in a variable, $loginurl, that you can use to create a hyperlink that points to Facebook and prompts the user to accept or deny the request for the specific permissions for your app. Assuming the user clicks “allow”, the user will be directed back to your website and you will now magically be able to query the information via the SDK. This $loginurl could replace your website’s “sign in” link if you wanted to simply force every user to have a Facebook account (and accept your app’s permissions). If a user has already given your application access to the data requested in the $loginurl url, Facebook will simply send the user to the redirect_uri, which should be your user dashboard/profile page. Some data, such as a user’s stream, is explicitly shown on a seperate permissions step on Facebook during the access granting process. A user will first allow you access to their bio data, and then they will allow or deny access to their stream. Should a user only allow access to their bio, and not their stream, the user will be bounced back to this permissions screen anytime your application attempts to access stream data. This actually integrates really nicely with web apps and keeps you from having to do constant error checking on data you’re asking for from Facebook.

Requesting Data

Below is an example of requesting the bio information:

//unique identifier on Facebook
$fbuserid = $facebook->getUser();

$user_profile = $facebook->api(‘/me’,'GET’);

$fname = $user_profile["first_name"];
$lname = $user_profile["last_name"];
$email = $user_profile["email"];  //this was a separate permission, remember?

Requesting wall/stream information:

$feed = $facebook->api(‘/me/feed’,'GET’);
for ($i=0;$i<count($feed["data"]);$i++)
{
// do something, like reading all shared links – $feed["data"][$i]["link"]
}

You could also check to see if a user has already liked your Facebook page:

$url = “/me/likes/$pageid”;
$like = $facebook->api($url,’GET’);
if($like["data"][0]["id"] != $pageid)
{
//do something, like suggest they like your page, or hide content until they do
}

This is just an intro into getting data from Facebook from PHP using the Open Graph API and Facebook PHP SDK. Facebook dogs and Google should be able to answer most of your questions.

Pages:123»