Time to Reevaulate Tindie & Paypal

This is a long post, but due to the severity of the situation, it warrants a full and thorough explanation. tl;dr : Paypal has a mission critical bug with their Adaptive Payments API that has been live for at least 9 days, and did not even know.

Today we received an email from a customer that was concerned his order wasn’t showing up under his account, yet it was visible on his Paypal account. After digging into our implementation of Paypal’s Adaptive Payments API, we didn’t find anything wrong. We did however notice we hadn’t received an IPN, an “instant payment notification,” from Paypal alerting us of the order. Anyone that uses Paypal as a payment processor depends upon IPNs for managing orders.

The process is:

  • Customer checks out of Tindie, redirect to Paypal
  • Pays on Paypal, redirect to Tindie
  • Paypal sends IPN to verify order was completed
  • Paypal sends another IPN with transaction fees

As Paypal says in their own documentation:

Merchants can use it to automate back-office and administrative functions, like fulfilling orders and providing customers with order status.

That is exactly how we use Paypal, and when there is a problem with IPNs, every merchant that depends upon Paypal has a monster issue because they aren’t being notified of customers paying for orders. Customers are paying, and companies have 0 idea there was a transaction.

I called Paypal Technical Support to verify exactly what we were seeing. Sure enough, they had received other calls today about this exact issue. Other vendors were not receiving IPNs for their orders. The ticket for this “critical bug” was created today.

The problem with that fact is that our customer’s order was from 9 days ago. For at least 9 days (we are going through all of our partial orders now to see the full extent of this bug), Paypal was not sending IPNs to merchants and did not know. This is a mission critical function for online businesses and we were never notified or alerted about it. The fact the ticket was made today also means Paypal didn’t know - and if they were unaware that such an important bug, then that brings into question their entire platform.

Because of how critical IPNs are to any Paypal integration, we didn’t think to double check if IPNs were not being sent. We are currently building that check as I type this. We are also going through older, partial orders to find any others that were affected, and resolve those as well.

Moving forward I have serious reservations about using Paypal due to the severity of this bug, and their lack of awareness in such a critical part of their system. We currently depend upon Paypal for our disbursements, but I am looking into other options. For US sellers, we could transfer funds through Stripe. Stripe lets us transfer funds to another US bank account or US Visa/Mastercard debit card. For international sellers, we’re open to other alternatives if you know of any.

I’m very concerned about our dependence on them moving forward. While I was on the call with Techincal Support, he said it had happened previously as well. This is at least the second time IPNs haven’t gone out, and I (along with countless other merchants) was unaware. Tindie is a small business, and we cannot have Paypal souring our reputation due to their ineptitude to provide a stable platform and fast, transparent support. The people I spoke to on the phone were great- it is Paypal’s policies which are the problem. The fact Paypal isn’t alerting merchants about this bug is a huge issue. How can we run our business when the platform we depend upon isn’t keeping us informed of their status? The answer is we can’t - and I’m unwilling to sacrifice our customers to Paypal’s ineptitude.

So how would you like to receive your disbursement? I’m open to exploring all options outside of Paypal.

Emile
Founder of Tindie

Wellcome to the internet! I would like to say that other payment vendors are better, but they are not.

What we do at work is keep Nagios checks for number of successful IPNs over time per vendor, and if we haven’t gotten X success in Y time, set an alert. Often, we are among the first to call in when vendors have problems, even some of the biggest in the industry! (VISA had a routing problem the other week for a while, for example.)

I applaud looking for more options. More than one is always good. However, I don’t think you can realistically drop PayPal just because you add some other option. Stay vigilant, and keep your options open!

Coinbase and Stellar seem like new disruptors in the payment space. Perhaps Stripe, too, although they have the credit card problems of chargeback and fraud and fees. Neither has the market reach of PayPal though.

1 Like

Hey Emile and Tindarians,

Some countries have very strict exchange control laws, and the banking system makes it very difficult and expensive to exchange currency. PayPal has actually made it allot easier for ZarDynamix as if there were no paypal and the sync up with FNB (First National Bank) then the only way to pay for any purchase would be via a bank draft which makes sales virtually impossible to motivate.

I get that PayPal may have hit a bump in the road, but please keep it in mind that for some of us it remains the least troublesome way of trading / operating :D.

Thanks in advance

Michael

Well, relying on IPNs for executing orders only doesn’t work unless the payment provider offers a permanent notification until your server acknowledges that. There are just too many problems which may happen, e.g. your server is unavailable, their IPN doesn’t work, you do an update while they send you an IPN, etc… Therefore we’re using also the Paypal transaction history API to synchronize the transactions every 5 minutes.

On July 16th, I notified PayPal of a critical issue with their IPNs. During my testing, I noticed that I simply wasn’t getting them. Looking at the IPN History in the account, I showed 9-12 attempts to send the IPN, but no success. After a couple of days of a very unhelpful back-and-forth with the PayPal support, and troubleshooting on my end, I realized this was because we were requesting the IPN over SSL. When I requested the IPN over non-SSL, it worked fine. I spoke with our sysadmin about this, he thought that it might be related to the SSL multiplexer we were using (sslh). After he changed this configuration, it started working fine over SSL. Though we found a fix for our needs, I still believe this is a problem on PayPal’s end (they’re not handling SSL properly).

Also, during this whole time, the Stripe webhooks were working flawlessly.

It could be something to do with the cipher suites supported on the paypal side not being compatible with the config on the server receiving the IPN request.

This reference is very similar but dates back a couple of years :

https://www.drupal.org/node/1801620