Wes Cutshall

If you don’t already know, Facebook has opened up their data for some, regulated, 3rd party use. Facebook calls this their Open Graph API (application programming interface). Any developer who follows the specified steps can create applications that integrate with Facebook user data. Facebook does apply limitations on how developers can use the data in order to protect Facebook users. But, for the most part, if a Facebook user agrees, you can gain access to just about all of their profile data, and even post to their wall, etc.

Getting Started

Before you can start developing with Facebook you’ll need to register an application (yet to be developed) with Facebook. Start on the Facebook Developers page and then go to the Apps tab. I’m not going to go into detail about this because it’s pretty straight forward and already written about everywhere on the web. To make getting to the data easier there is a PHP SDK for Facebook Open Graph. We will be using this library for all of our Facebook queries.

Permissions

Before you can get to any data a user is going to first have to grant your application access to specific parts of their data. Facebook has different data and actions sectioned into permission groups. Below is an example of requesting permissions to get a user’s email address, bio data (name, etc), and to be able to read their wall.

$config = array(
‘appId’ => FBAPPID,
‘secret’ => FBAPPSECRET,
);

$facebook = new Facebook($config);
$fbuserid = $facebook->getUser();

$params = array(
“scope” => “read_stream,email,user_about_me”,
“redirect_uri” => “http://www.wescutshall.com/”
);

$loginurl = $facebook->getLoginUrl($params);

This will result in a variable, $loginurl, that you can use to create a hyperlink that points to Facebook and prompts the user to accept or deny the request for the specific permissions for your app. Assuming the user clicks “allow”, the user will be directed back to your website and you will now magically be able to query the information via the SDK. This $loginurl could replace your website’s “sign in” link if you wanted to simply force every user to have a Facebook account (and accept your app’s permissions). If a user has already given your application access to the data requested in the $loginurl url, Facebook will simply send the user to the redirect_uri, which should be your user dashboard/profile page. Some data, such as a user’s stream, is explicitly shown on a seperate permissions step on Facebook during the access granting process. A user will first allow you access to their bio data, and then they will allow or deny access to their stream. Should a user only allow access to their bio, and not their stream, the user will be bounced back to this permissions screen anytime your application attempts to access stream data. This actually integrates really nicely with web apps and keeps you from having to do constant error checking on data you’re asking for from Facebook.

Requesting Data

Below is an example of requesting the bio information:

//unique identifier on Facebook
$fbuserid = $facebook->getUser();

$user_profile = $facebook->api(‘/me’,'GET’);

$fname = $user_profile["first_name"];
$lname = $user_profile["last_name"];
$email = $user_profile["email"];  //this was a separate permission, remember?

Requesting wall/stream information:

$feed = $facebook->api(‘/me/feed’,'GET’);
for ($i=0;$i<count($feed["data"]);$i++)
{
// do something, like reading all shared links – $feed["data"][$i]["link"]
}

You could also check to see if a user has already liked your Facebook page:

$url = “/me/likes/$pageid”;
$like = $facebook->api($url,’GET’);
if($like["data"][0]["id"] != $pageid)
{
//do something, like suggest they like your page, or hide content until they do
}

This is just an intro into getting data from Facebook from PHP using the Open Graph API and Facebook PHP SDK. Facebook dogs and Google should be able to answer most of your questions.

I haven’t done real well keeping up with posting interface design issues I find on websites/software, but here’s one. It’s generally bad to have the same noun/verb/command name in two different places, especially when they do two different things. When creating an email in Thunderbird, the new email menu bar has an “Options” dropdown as well as an “Options…” item under “Tools”. This is not optimal and should be avoided to prevent confusion; “It’s under Options”.

Tweeting to your account from PHP. I’m fresh off of adding this feature to another one of my sites so I thought I’d document the process for people as it’s not so obvious. I post important events to Twitter that happen on the website based on user interaction. I do not use this to allow users to post to THEIR Twitter accounts. So I’m posting events like “New Lost Dog Added” or “New Naming Assignment”, etc. So I’m only posting to MY Twitter account.

1. Download the TwitterOAuth library for PHP from http://github.com/abraham/twitteroauth/downloads and upload it to your website in a new directory /twitter.
2. Visit https://dev.twitter.com/ and click Create an App. You will be asked to sign in as the Twitter account you will bill posting to.
3. The Create an App wizard is pretty straight forward, except that the default rights are Read-Only. You have to go back and edit the rights to be Read/Write and resave the settings.
4. Back on your website in the Twitter library code you will need to copy/paste the correct keys into the twitter/config.php file. You get these keys from the Twitter developer page for your new application. Twitter allows you to generate the keys on the dev site that you need for the OAuth keys on the config file. You have to click a button on the bottom of the details page to create the OAuth keys you will need. Note that if you change the application permissions (Read-Only, etc) the OAuth keys change and need to be re-copied. So copy the 4 keys to the appropriate constants in the config file.
5. Also edit the callback URL to be example.com/twitter/callback.php.
6. Load the twitter/index.php page from your website in your browser, click Sign In, and see that you get a long response from the API with a bunch of text that you don’t need to read. The error responses are short and sweet, so if you get a full array of data as a response then it signed in correctly. Now you know your sign-in works, just time to set up tweeting. If this step doesn’t work then re-check the config file, make sure you have the correct callback URL, etc.
7. Create a function for tweeting:
   function tweet($text)
   {
   include(‘twitter/twitteroauth/twitteroauth.php’);
   include(‘twitter/config.php’);

   $connection = new TwitterOAuth(CONSUMER_KEY, CONSUMER_SECRET, OAUTHTOKEN, OAUTHSECRET);

   $content = $connection->get(‘account/verify_credentials’);

   return $connection->post(‘statuses/update’, array(‘status’ => $text));
   }

8. Now add tweet($text) at important parts in your PHP code. Make sure you’re tweeting relevant information about events on your website.

Thanks to Stumbleupon, I discovered this service about a week ago. ClickTale utilizes JavaScript cursor position (X/Y coordinate) tracking to generate both website user interaction “videos” and interaction maps for your website. Where Analytics can tell you a navigation path (page A, then page B, then page C), ClickTale can show you exactly what a user did on a page in between the clicks. You can replay a user’s visit to your website. You see the content of your website with a cursor moving over the site just as the user navigated the website. You can see mouse movement, clicks, page scrolling, and in some cases the characters that were typed into form fields (this only works on some recordings, so I assume it’s browser or OS-dependent, the majority of recordingss showed “?” instead of the exact character) in real time. This service can be very useful for solving website interaction issues, some that you don’t even know you have.

If your conversion rate is low you can see exactly where users quit using your registration or order form. If only JavaScript could also record audio it would be just like being there (“Ahh, too expensive.” <exit>). I tested ClickTale on Naming Force and found it very intriguing to watch people using the website. I focused on users who visited the New Assignment page. You can narrow down the recordings to view users who viewed a certain page, among other parameters. The recordings start as the users entered the website, so you get to see how they navigated to the page in question, as well as what they did on that page.

One of the most shocking things I saw was that most of the users scrolled all the way down the home page, when the home page was the site entry point. With so much talk about Above the Fold I was a bit surprised that the content was engaging enough that most of the users scrolled down the whole home page before using any of the navigation items. Pleasant surprise. As far as conversions on the New Assignment page, I haven’t yet found anything that has lead me to modify the form. Many users visit the page briefly without scrolling, which shows that they simply are checking the site out without much interest in submitting an assignment (yet). Very few users started to fill out the form only to exit the page. Of the users that started to fill out the form and then exited, most of them (less than a handful with my small one-week sample size) seemed to do so after viewing the (very affordable) prices. This has brought up the question of whether the package options should be moved to the top of the page, but in regards to conversions, I don’t see how this would matter one way or the other. The only interface change that has been made at this point due to watching ClickTale videos happened to be on a somewhat unimportant page. But, it revealed some interesting UI issues. Naming Force has a Random Name Generator for users to use. It consists of a large white box with a button underneath. You click the button, repeatedly, to generate random names that appear inside the white box. I watched one user visit this page and click 20-30 times inside the white box, in various spots, never clicking the bright orange button underneath, only to (I assume, frustratingly) leave the page. So, I added “click button below” inside the white box, as somehow it wasn’t obvious to at least one person. Hopefully I’ll find enhancements I can make to some of the more important pages based on these interaction videos.

One of the other features of ClickTale is the Heatmaps. Heatmaps take all the data collected, mouse move, click, scroll, etc and chart them on an overlay of your website. This gives you an idea of where people are concentrating their focus on your pages. Mouse move and click are pretty straight forward, you can see a heatmap of the most concentrated areas (see image below). Scroll reach is another heatmap that might need some explanation. This heatmap maps the percentage of users who scrolled down where the part of the page was in the browser window. This is where the Above the Fold theory can be tested for your site without watching hundreds of videos.

Mouse Move and Scroll Reach Heatmaps

Overall the ability to record user mouse movement and play it back is pretty incredible. The only problem I see with ClickTale is that the prices are a bit unrealistic for the majority of us smaller website owners. In order to view details for pages that are not my top URL on Naming Force, for example, I had to sign up for the subscription that is $290 / month. I don’t see myself continuing this service past a month, which is a real shame for ClickTale. I’d happily pay $30-$50 a month, but I can’t find $290 worth of modifications to make each month to make the subscription worth sticking with. I feel like this service’s business model should be to get users to sign up for an insignificant amount and allow them to “forget” that they have the subscription, or technically just to not care enough to cancel it and allow it to recharge them for years. Users won’t want to cancel a $50/month subscription, just in case they want to look through the collected data in the future, and ClickTale could make a lot more through long-term subscriptions, IMO.

After reviewing the failed email queue that was 100% full of Comcast email addresses I decided it was yet again time to review the email/DNS settings to see if there was something I had missed that was preventing Comcast from allowing emails from ALL of our sites from getting through. Comcast does have a set of web pages dedicated to email errors and failed email solutions. Although I didn’t know exactly why Comcast was blocking emails, I decided to fill out the Blocked Provider Request form that allows you to submit your mail server IP address and ask for a ban to be removed. The page says that Comcast will usually respond in 30 minutes, and much to my surprised less than 30 minutes later I received an automated email saying that first of all, yes, the IP had been blocked, and that the block was now removed! The email did say that the IP had been blocked for “patterns characteristics of spam”. Hopefully the previous DNS settings that weren’t 100% up to par caused this block and it won’t return. About a week or so ago I found some DNS settings that needed to be modified based on some reports I ran on DNSStuff.com. Hopefully that will help Comcast to keep us on their white list.

What is it?

In short, jQuery is a JavaScript library that makes it easier to do some of the things you do in JS every time you create a new website. If you adopt jQuery for even one website you will end up adopting it for every website after that.  JQuery changes the way you select and modify HTML elements in JavaScript.

I’ve never been big on JavaScript libraries, since I feel that I can just about write anything in JS that I need. Although I will Google for code from time to time, I’ve just never found much of a need for a JS library. I will admit  that I’ve used Yahoo’s drag’n'drop controls on LeagueAce.com, though. I had heard of jQuery, or at least seen mentions of it in forum posts, but never tried to discover what it was. Then one day, when I had absolutely nothing to do, I Googled “jQuery”. Now, jQuery’s website isn’t the best place to figure out what it does, IMO. I had to read a few blog posts and then read through the jQuery documentation before it made any sense. I think that jQuery could do a much better job of explaining what it is to newbies on their website.

JQuery Example Usage

The most common use of jQuery is to easily select and modify HTML objects in JavaScript. Please note that the following code examples are not tested, are from my memory, and might need slight changes to actually work. Where we normally code “document.getElementById(element).className = class” we can now code “$(element).addClass(class)“. Where we normally have to write a loop in order to get the value of a selected dropdown item with standard JS, in jQuery we can instead code “var test = $(element).val()“. JQuery simplifies the process of selecting the correct HTML element, changing its properties, getting it’s value, etc. JQuery allows you to select elements in different ways, which is one of the coolest features. You can select an element via it’s HTML ID ($(“#id”)), via the tag type ($(“:input text”)), via CSS class ($(“.class”)), etc. JQuery can also select elements based on parent-child relationships. So you could select every text input element inside of a DIV with the ID of “divTest” ($(“:input text”, “$divTest”)), for example. As mentioned in the dropdown menu example jQuery has foreach ability for looping through elements that are selected. JQuery allows you to string together methods so you could make multiple changes to one object by simply adding “.method1(value1).method2(value2)” to the end of your command.

Summary

JQuery has some other features, like transition effects, that are “nice” but far from the reason why I started using it on all my new websites. The simplicity of how you can select HTML elements makes you wish that JavaScript had been written like this from the beginning.

What is a Feedback Loop

Some ISP’s and email providers allow website owners to register into Email Feedback Loops. A feedback loop allows you to register to have emails that users mark as Spam forwarded to a specified email address. Each network requires different levels of information to register. I have registered with Yahoo’s Feedback Loop for all of my websites. I believe that Yahoo requires that you have DKIM set up to register (remember seeing it mentioned,  already had it set up, didn’t affect me).

The Painful Truth

What I discovered through signing up for the feedback loop was bothersome. Be aware that you might become extremely frustrated if you sign up for a feedback loop and discover that your users are marking your opt-in emails as Spam. Imagine you run the largest Lost Dogs website in the world and users are marking “found dog added” customized alert notification emails as Spam instead of signing in and modifying their account after their lost dog is returned. Emails from a website that is trying to reunite lost pets, that people signed up for, is being marked as Spam by it’s own users, for no good reason. We’ve had deliverability issues from time to time with Yahoo, most of which have been resolved with the techniques in this blog post. But I would bet that these users recklessly marking emails as Spam contributes to our IP address being marked as a bulk mailer with Yahoo.

Making Lemonade

But you have to take something away from what you find is being marked as Spam. What I noticed was that most of the emails marked as Spam were for lost dog listings from 2008 or earlier. Currently we don’t ever expire alerts, people have to turn them off, but that’s going to change. We just can’t afford to have emails marked as Spam. Now, Yahoo tells you that having one email marked as Spam doesn’t affect you, but with a handful every other day I don’t see how Yahoo isn’t using this information against us. And I’m using this Yahoo feedback loop to extrapolate what is probably happening on other networks which might not be so forgiving.

Here is a list of some feedback loops that you might want to register for. At least register for one of the major ones and get an idea of what people are marking as Spam that you send out.

One of the biggest problems I’ve run into as a commercial website owner is consistently delivering emails generated by my websites. The issues in doing this have been various and at each step of the way I’ve learn something that helps us to to deliver messages there on out.  But each lesson comes from a shortcoming. Hopefully I can spell out some of the things I’ve learned that have helped to increase email deliverability for our websites.

The Number One Problem

The number one problem that affects email delivery is the server from which the emails originate. Or that server’s “reputation” at the time that you send your emails. If you run your website on a shared hosting environment, you will have problems getting emails delivered consistently to your users. When you have no control over what emails are coming from your email server’s IP address you put your own deliverability results in jeopardy. One of the misunderstandings we had along the way was that having a dedicated web server meant that we had a dedicated email server. Like many programmers we simply used PHP’s mail feature and never really knew where the emails went from there. Although our website was delivered from a dedicated server, all outgoing emails were routed through a small collection of email servers. It’s not a problem that your email server has a different IP address than your website, it’s just an issue that you can’t keep Joe Schmoe from sending spam, or just sending too much email – which looks like spam, through “your” (shared) email server. The webhost in question did not allow us to run our own email service on our web server, so we had problems using their email server until we later moved to Rackspace. I’m a programming, and not a network admin guy, so I don’t know even know how to set up a Linux/Unix web server to send mail, I was just told it was not allowed with our webhost, even though we had a dedicated server. With this host it was much more of a managed server, and we could do a lot less than we can do now at Rackspace. Not that this type of hosting bad, it’s just not ideal for what we needed. So during the time that we had a dedicated web server but not email server we ended up using a 3rd party email service provider (ESP) to help get emails delivered more consistently. This definitely helped, complaints were down, but it wasn’t the ideal solution, and costs more than sending emails strictly through your own server. The very best scenario is to either have your own separate email server (overkill) or have your web server box configured to be your own email server. This way you have control over your server’s reputation with the ISP’s and email providers of the world.

Reply To?

This tip is one that seems a little ridiculous, but I tested it and it’s completely true. When generating emails programmatically I never thought to populate the “ReplyTo” field. I just thought that email clients reply to the From address unless the ReplyTo happens to be filled out. So why would I ever put something in the ReplyTo property? It turned out that with one major email service provider if I sent an email without the ReplyTo specifically set to the same address as the From field, it would send the email to the spam folder. Weirdest thing to me, but when I discovered this I just changed all of my code to add the From address as the ReplyTo address. This kept emails from going to the spam folder for this one major email provider.

DNS Entries

A few things at the DNS level can be done to help ensure email delivery. I don’t know these settings, or entries, well enough to deeply discuss setting them up, since I set them up once with Rackspace based on articles I read, and haven’t touched them since, so I’m just going to touch on the concepts to guide you in the right direction.

1. One of those settings is reverse DNS lookup entry. It’s one thing for Example.com to resolve to 98.129.58.195 when you attempt to visit the website, but if an email server gets emails from 98.129.58.195 that claim they are from example@example.com, how can it test to see if the sending server really hosts Example.com? Reverse DNS lookup. I won’t attempt to discuss the technical details, because I don’t know them, but be sure that your DNS records have reverse DNS set up so that email servers can verify that your emails are coming from the correct server.
2. Another DNS record that can help email delivery is an SPF (Sender Policy Framework) record. The idea behind an SPF record is for your domain server to be able to be queried and asked for a list of email servers that are allowed to send email for your domain. Even if you’re sending emails from the same server IP address, it helps to project to the email servers out there that your emails are allowed to be created from this IP address for your domain name.
3. The third DNS record that can help with delivery is DomainKeys. This was a little more complicated to set up, but luckily Plesk took care of all the hard work. DomainKeys is a more sophisticated way of identifying that a specific message was sent from a specific server. DomainKeys works in a public-key scenario, so a key is sent in the email that can be verified by querying the DNS server. Google it, or contact your webhost to find out how to set this up. It will be a combination of server settings and DNS entries.

Abuse@ and Postmaster@

Another thing that email servers can detect is the existence of certain email addresses. It’s a common tip that having a valid abuse@example.com and postmaster@example.com helps to deliver emails. This one I can’t vouch for, but I do have them set up because I’ve read it multiple times.

Handle Bounced Emails

One thing that can make you look like a spammer is sending emails to bad email addresses. Whether they never existed or are expired, it just looks bad. You can minimize this by handling bounced emails correctly to keep for resending emails to bad addresses. The way we do this is to:

1. Set the Return-Path value in your email headers that contains the email address being sent to for each email. In doing this your giving yourself a way to know where an email was sent when it bounces back to your saver. An example of what you can put in the Return-Path header field is user=yahoo.com@example.com. It’s the user’s email address in front of the @example.com with the “to” address’s @ sign replaced by “=” to keep it valid. So an email to user@yahoo.com has a return path of user=yahoo.com@example.com.
2. Have a default email address handle all mail to invalid email addresses for your domain (user=yahoo.com@fidofinder does not exist). So we have all emails sent to an invalid address funnel to bounce@example.com.
3. Pipe the Return-Path “address” to a script that can process them, most likely adding them to a database table of emails to not send to. I have a PHP script that can do this for qmail. Contact me if you need it. It just reads the email left of the “@” and replaces the “=” and puts it in a table. The code is a little weird as it has to read the stream of data being piped to it. It took me a while to find the correct code to do this.
4. Obviously, you now consult the database table when sending emails to stop sending emails to addresses that have bounced in the past. This again just prevents you from looking like a spammer.

Timing and Volume

Two things that affect if your valid email looks like spam is the timing of the email and the volume. Large email providers know if you are sending the same email to thousands of addresses. And even if this is to registered users of your website, it can look spammy. Avoid this by sending emails in spurts. If you need to email 100K users write a script to do this a little at a time throughout a day. And avoid sending emails between midnight and 5 AM, even though it’s the choice of most programmers to do maintenance tasks at low-volume times. Spam too is mostly sent during these hours. Another tip is that if you want your emails to not appear to be spammy, send more emails on a regular basis. I’ve always felt that sending fewer emails would be better, but I’ve seen suggestions of the opposite.

Unsubscribe and Postal Address

They say that the major email providers can tell if you included an unsubscribe link at the bottom of your email or not. Whether they actually do this I do not know, but since I’ve read that they do, I include it. Don’t assume that the URL has to go to a fancy automatic unsubscribe system. It’s fine if it goes to a page where a user can sign in to change their settings or delete their account. And since in some states it’s required to have a postal address included in email marketing, throw your postal address at the bottom as well. I’ve read that this too is detected by the major providers.

That’s It!

Well there you have it. There are probably other tips, but those are the ones I’ve learned over the past 6 years. It’s been a huge learning process, and a thorn in my side at times, but currently emails are delivered at a fairly good rate which I can only measure by a low volume of user complaints.

I recently had to reinstall my OS, which led to me upgrading to Windows 7 (from Vista), which led to me reinstalling all of my software, which led to me upgrading some of my software. I decided to purchase and download the latest version of Zend Studio which I use for PHP development. It’s not cheap, $399, but I use it often enough that I decided if I can make my PHP development IDE better, it’s worth it (I was wrong).

Zend Studio 7 is Broken

I download and install 7.x, I previously had 5.5, and at first I’m just out of my comfort zone because it’s different. I decide to just learn the new interface and spend the next few hours getting it set up with my websites. I remotely develop via FTP. In 5 I was able to simply add the FTP servers to my left-hand pane and work with the files. You can still do this in 7, but they removed some of the features they had in 5. In 5 I could set the Initial Directory of my FTP servers. If I do not set this I have to expand 2-3 directories before I get to the actual root of my webserver. It’s not only a pain to do when you work on multiple servers, but it also increases the width that the content of my left-hand pane, making me have to scroll to the right to work with files, and back to the left to open another server. It’s just not ideal, and it’s not the same, or better, than 5.

I put up with this inconvenience and started to write some code. I had a programming task I wanted to complete and decided to jump in and perform it. As I started to type in Zend Studio 7 there was a significant delay in the screen response. As I typed the cursor was 2-3 letters behind. Because I’m a programmer, I know what is going on here. Zend is validating what I’m typing, and not doing it very well. I spent some time in the settings and turned off all of the validations (that I could find). It got better but there was still a delay, enough to bother me but I moved on. Then I noticed that when holding down the backspace key nothing happened. I could not hold down backspace and delete a few words or a complete line. The cursor would disappear and nothing would happen. I had to hit backspace over and over and over to delete a word or two.

No Refunds

I accepted that this wasn’t going to work out, and that I was 100% happy with Zend Studio 5, so I should just downgrade and give up on 7. I uninstalled 7, reinstalled 5, and submitted a refund request on the Zend website, and that’s where it got hairy. In an email response from Orly Maman from Zend I was told that since Zend offers a 30 day evaluation version and I had the opportunity to review the new version of the software.  Because of this there are no refunds. You’ve got to be kidding me. Who in their right mind is going to test-drive an upgrade of their current software? No one. I responded that this was not acceptable, that I was not ever going to use 7, had already downgraded, and that I just wanted a refund. Same reply. So I decided to call the main U.S. office. On the phone I got the same reply. I was told that I should have read the user agreement which says there are no refunds for any reason. Completely ridiculous.  I don’t even have to expand here how horrible it is for a company to operate this way, so I’ll leave it at that. Needless to say I submitted a chargeback with my credit card and the money will be returned in the end but Zend has lost a customer for sure.

Orly MamanOrly Maman

I recently ran into a problem with a 3rd party JavaScript source call slowing down the load of one of my sites. I decided to change the code to perform a document.write of the external source tag when the page’s initialize JS code ran. Apparently the 3rd party developers wrote code to test for the correct insertion of their code, and whenever I made this change the script suddenly stops working, and instead pops up a JS alert saying that you need to place the code between the “<body></body>” tags. Their validation code is erroneous, but that doesn’t help me. I ended up finding a solution to my problem in the following script I found online. Somehow adding the code dynamically do the head tag doesn’t trigger the same warning, and the script still works.

var head= document.getElementsByTagName(‘head’)[0];
var script= document.createElement(‘script’);
script.type= ‘text/javascript’;
script.src= ‘http://www.example.com/script.js’;
head.appendChild(script);