David Cummings on Startups

5 Challenges for Automation in the API Economy

Written by

David Cummings

in

, ,

TechCrunch has a piece up today by Alex Williams titled Speed and Automating the Connections Between Humans and Machines in the API Economy. In the article, Williams argues that speed of an API, especially under large load, is a real challenge, just like scaling a large website (they are in fact very similar with APIs potentially having more write load than read load, in some cases). In addition to speed, he highlights automating the connections between APIs as a challenge, where automating means integration and connection of disparate systems.

Peeling back API automation to a more detailed level, here are five challenges I see:

  • Data Interoperability – Synchronizing data between different systems is challenging due to different standards in types of data allowed (e.g. challenges with date/time stamps, number of characters allowed, translating fields like ‘GA’ to ‘Georgia’, etc)
  • API Authentication – While there are standards like OAuth and OAuth 2.0, many APIs were built before the standards were established and have their own form of authentication, requiring more effort to integrate as well as more ongoing maintenance
  • Recent Data Polling / Ping Backs – To connect disparate systems there’s a requirement to constantly check for recent data, or set up a ping back to be notified of new data, only many systems are still immature when it comes to this functionality by simply returning all data or only returning data in a paginated form (instead of being able to query against a specific data/time)
  • Bidirectional Syncing – It’s fairly straightforward to set up one-way syncing where one system is the master and the other system only takes, but doesn’t give data. Things become much more complicated when true bidirectional syncing is required and data can flow either way between system.
  • Custom Fields / Ad Hoc Customizations – Many of the more powerful systems, including Salesforce.com, allow for infinite customization, which makes for more complexity when trying to integrate products.

The API economy is going to be a major driver of innovation over the next 5 – 10 years, and getting the automation piece right is a big opportunity.

What else? What are some other challenges for automation in the API economy?

Comments

2 responses to “5 Challenges for Automation in the API Economy”

  1. Jay Myers Avatar
    Jay Myers

    Here’s a few more I see as potential challenges we need to solve:

    Data APIs that allow deep discovery — There are a ton of APIs spewing large amounts of data for users to consume. But what does that data really mean? Is it queryable? Can I form complex queries that truly allow my end user to dive deep into the my data and unearth new, interesting, impactful things?

    Adoption of (some) standards to ease friction between systems exchanging data — Not all of us see the world in the same way, and we certainly don’t define our data in the same way. However, there are standards available for use in APIs and data exchange that would allow base “rulesets” (commonly referred to as Vocabularies/ Ontologies) to define attributes that are common in the data objects we model. One example that is already having major impact is the GoodRelations vocabulary, used to model products and product data. This vocabulary has served as the base for the schema Google, Yahoo and Bing use on schema.org to define products. By using a bit of structure, we enable machines to better understand our data and do amazing things with it. There’s no reason API creators couldn’t make their APIs more understandable to other systems by including a bit more structure/ standards in their APIs.

    Real time data — Going a bit deeper on the polling/ ping back concept, many systems (even large corporate ones!) are unable to keep up with the sheer amount of changes the data goes through on a daily…err… more like hourly or every half an hour basis. It’s not that they can’t respond in a timely fashion, but that the data is actually changing and correct. Let’s take a retailer for example: product pricing and availability is huge in the industry. If the pricing isn’t up to date or the quantity shows ‘1’ when there really is ‘0’, it can lead to massive customer disappoint. How do API builders insure correct, timely data not just in their system, but in all downstream systems that consume it?

    My two cents 🙂

    Reply
  2. Josh Avatar
    Josh

    Interesting article. We are working to solve these and other challenges with APIs at maasive.net (an Atlanta based startup). Just had our first customer in the private beta go live a few weeks ago.

    Reply

Leave a reply to Jay Myers Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

More posts