Streaming API

Estimated time: 25–35 minutes

This tutorial will step you through a project to introduce you to the concept of streaming data. You will livestream real-time data from the Twitter API and extract tweets referring to either Tesla or Netflix.

[info] What is a streaming API?

A streaming API is one that is event-driven. This means you do not need to poll the API to get new information. When new information arrives (for example, a new tweet is tweeted), the API will send out the new information. Using nio, you will subscribe to Twitter's streaming API using the Twitter block as a generator block to drive the service with data.


Create a Twitter app

[info] What you will need:

  • Twitter account
  • Mobile phone number
  1. Visit the Twitter Developers Site.
  2. Sign in with your Twitter Credentials.
  3. Click Create New App and fill in the following Application Details:
    • Name: nio-workshop-<your name here> (this needs to be unique).
    • Description: A test application for nio services.
    • Website: https://www.placeholder.com.
    • Leave the Callback URL field blank.
  4. Accept the terms of the Developer Agreement.
  5. Click Create your Twitter Application.
  6. Click the Keys and Access Tokens tab.
  7. Click Create my access token.
  8. Make a note of the following tokens:
    • Consumer Key (API Key)
    • Consumer Secret (API Secret)
    • Access Token
    • Access Token Secret

Add credentials to your instance as environment variables

Your keys and access tokens are unique identifiers of your Twitter account and should be stored as environment variables rather than hardcoded into blocks. See the documentation on environment variables for more information.

  1. Select your Tutorials instance in the left navigation panel and click the edit button in the toolbar.
  2. Click the + environment variable button four times to create a key:value pair for each of your Twitter keys/secrets, and enter each of the following:
    • key: TWITTER_ACCESS_TOKEN
      value: <your Access Token>
    • key: TWITTER_ACCESS_TOKEN_SECRET
      value: <your Access Token Secret>
    • key: TWITTER_API_KEY
      value: <your Consumer Key (API Key)>
    • key: TWITTER_API_SECRET
      value: <your Consumer Secret (API Secret)>

 


Create service

This service is going to obtain tweets that mention either Tesla or Netflix and append a count to each tweet coming in from the API.

  1. Click create new service.
  2. In the service name box, enter tweets-count.

Add preconfigured Logger block

Reuse the Logger block you configured in the RESTful API tutorial.

  1. In the configured tab of your block library search for Logger.
  2. Drag the preconfigured Logger block to the canvas.
  3. When the block reuse warning displays, click got it!

[info] Block reuse

When you reuse a configured block, any configuration changes made to the block will be made across all instances of that block.


Add and configure Twitter blocks

The nio Block Library has a variety of Twitter blocks. In this service, you will use the block type called Twitter.

You will configure the two Twitter blocks you will add next to emit a signal every time someone tweets about either Netflix or Tesla. In order to connect to the Twitter API, you will also need to configure the blocks with your credentials from the app you created in the previous step.

The block configuration for the Twitter blocks will look similar to the image on the right. Use double brackets to indicate environment variables. Environment variables will be replaced with their configuration values when the service is started, but the block's configuration will always contain the environment variable names.

  1. Download and install the Twitter block to your instance.
  2. Drag a Twitter block onto your canvas.
  3. Configure the block as follows:
    1. Name: Get Tesla Twitter
    2. Click + Query Phrases, enter Tesla
  4. Click accept.
  5. Drag a second Twitter block onto your canvas.
  6. Configure the block as follows:
    1. Name: Get Netflix Twitter
    2. Click + Query Phrases, enter Netflix
  7. Click accept.

Add and configure Counter block

In order to determine which company is receiving the most tweets, you need to count the number of tweets that reference each company. The Counter block will increment the cumulative_count every time a signal passes through the block.

  1. From the available tab of the block library download and install the Counter block.
  2. Drag a Counter block onto your canvas.
  3. Configure the block as follows:
    1. Name: Count Tweets
    2. Load from Persistence: false (deselect radio button)
    3. Exclude Existing: false (deselect radio button)

The Counter block group

The Counter and CounterFast blocks are part of a single repository. When you install the Counter block, the CounterFast block will also be installed.

Add configured block

  1. Click on the configured tab in the block library.
  2. Search for Counter in the search box.
  3. Drag a second instance of the Count Tweets block onto your canvas.

[info] Block help

From within the block configuration window, click the question mark at the top of the window to display the details of the block. Read the description to understand the difference between the count and cumulative_count attributes. You will use cumulative_count later in this tutorial.

 

[info] "Load from Persistence" and "Exclude Existing" options

Load from Persistence determines whether certain attributes in the output signal should persist on a service restart. Since you set the value to False (deselected the radio button) in the Count Tweets block, the cumulative_count attribute will reset every time you start the service.

Exclude Existing specifies whether the input signal's attributes should be excluded from the output signal's attributes. Since you set the value to False (deselected the radio button), all of the Twitter data will still be part of the signal and the Counter block's attributes will be appended to it. After completing the next step, feel free to select the radio button and see how your output changes. Just make sure the box is deselected before moving on to the next part of this tutorial!


Connect blocks and start service

You can now view the Twitter data coming from the API! After the next few steps, you will see Twitter streaming API data with an appended count attribute in each signal in the logger panel.

Blocks, such as the Twitter block, can have multiple input and/or output terminals for signals to flow through. Signals are routed to particular output terminals depending on the logic within the block and the input terminal that the signal entered the block through.

  1. Connect the blocks according to the image at right.
  2. Click save.
  3. Click start.
  4. Open the logger panel to view the Netflix tweets.

Each signal will have the cumulative_count in addition to all the data about a tweet from the Twitter streaming API in a format similar to what is shown below:

 

Click arrow to collapse/expand signal
{
  "contributors": null,
  "coordinates": null,
  "count": 4,
  "created_at": "Fri Sep 15 19:03:11 +0000 2017",
  "cumulative_count": 1379,
  "entities": {
    "hashtags": [],
    "symbols": [],
    "urls": [],
    "user_mentions": [
      {
        "id": 156293122,
        "id_str": "156293122",
        "indices": [
          3,
          14
        ],
        "name": "David Osler",
        "screen_name": "finance_LL"
      }
    ]
  },
  "favorite_count": 0,
  "favorited": false,
  "filter_level": "low",
  "geo": null,
  "group": null,
  "id": 908768175348764673,
  "id_str": "908768175348764673",
  "in_reply_to_screen_name": null,
  "in_reply_to_status_id": null,
  "in_reply_to_status_id_str": null,
  "in_reply_to_user_id": null,
  "in_reply_to_user_id_str": null,
  "is_quote_status": false,
  "lang": "en",
  "place": null,
  "quote_count": 0,
  "reply_count": 0,
  "retweet_count": 0,
  "retweeted": false,
  "retweeted_status": {
    "contributors": null,
    "coordinates": null,
    "created_at": "Thu Sep 14 09:09:54 +0000 2017",
    "entities": {
      "hashtags": [
        {
          "indices": [
            120,
            127
          ],
          "text": "Brexit"
        }
      ],
      "symbols": [],
      "urls": [],
      "user_mentions": []
    },
    "favorite_count": 19538,
    "favorited": false,
    "filter_level": "low",
    "geo": null,
    "id": 908256479932829696,
    "id_str": "908256479932829696",
    "in_reply_to_screen_name": null,
    "in_reply_to_status_id": null,
    "in_reply_to_status_id_str": null,
    "in_reply_to_user_id": null,
    "in_reply_to_user_id_str": null,
    "is_quote_status": false,
    "lang": "en",
    "place": {
      "attributes": {},
      "bounding_box": {
        "coordinates": [
          [
            [
              -0.06111,
              51.549969
            ],
            [
              -0.06111,
              51.648542
            ],
            [
              0.023642,
              51.648542
            ],
            [
              0.023642,
              51.549969
            ]
          ]
        ],
        "type": "Polygon"
      },
      "country": "United Kingdom",
      "country_code": "GB",
      "full_name": "Walthamstow, London",
      "id": "26b0db32cfda0432",
      "name": "Walthamstow",
      "place_type": "city",
      "url": "https://api.twitter.com/1.1/geo/id/26b0db32cfda0432.json"
    },
    "quote_count": 342,
    "reply_count": 244,
    "retweet_count": 8274,
    "retweeted": false,
    "source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
    "text": "I'm going to cancel Netflix and negotiate with each film producer separately, to get the best deal for me and my family #Brexit",
    "truncated": false,
    "user": {
      "contributors_enabled": false,
      "created_at": "Wed Jun 16 14:55:18 +0000 2010",
      "default_profile": false,
      "default_profile_image": false,
      "description": "I cover insurance, finance and law for @LloydsList, All views personal, Retweets not endorsements.",
      "favourites_count": 1326,
      "follow_request_sent": null,
      "followers_count": 2500,
      "following": null,
      "friends_count": 1019,
      "geo_enabled": true,
      "id": 156293122,
      "id_str": "156293122",
      "is_translator": false,
      "lang": "en",
      "listed_count": 84,
      "location": "London",
      "name": "David Osler",
      "notifications": null,
      "profile_background_color": "FFFFFF",
      "profile_background_image_url": "http://pbs.twimg.com/profile_background_images/431798723140526081/pEb72x3d.jpeg",
      "profile_background_image_url_https": "https://pbs.twimg.com/profile_background_images/431798723140526081/pEb72x3d.jpeg",
      "profile_background_tile": true,
      "profile_banner_url": "https://pbs.twimg.com/profile_banners/156293122/1455270383",
      "profile_image_url": "http://pbs.twimg.com/profile_images/892404701290999809/08q2c9qL_normal.jpg",
      "profile_image_url_https": "https://pbs.twimg.com/profile_images/892404701290999809/08q2c9qL_normal.jpg",
      "profile_link_color": "001C53",
      "profile_sidebar_border_color": "FFFFFF",
      "profile_sidebar_fill_color": "DDEEF6",
      "profile_text_color": "333333",
      "profile_use_background_image": true,
      "protected": false,
      "screen_name": "finance_LL",
      "statuses_count": 5212,
      "time_zone": "London",
      "translator_type": "none",
      "url": "http://www.lloydslist.com",
      "utc_offset": 3600,
      "verified": false
    }
  },
  "source": "<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>",
  "text": "RT @finance_LL: I'm going to cancel Netflix and negotiate with each film producer separately, to get the best deal for me and my family #Br…",
  "timestamp_ms": "1505502191756",
  "truncated": false,
  "user": {
    "contributors_enabled": false,
    "created_at": "Sat Feb 04 18:23:03 +0000 2012",
    "default_profile": true,
    "default_profile_image": false,
    "description": null,
    "favourites_count": 14,
    "follow_request_sent": null,
    "followers_count": 145,
    "following": null,
    "friends_count": 1568,
    "geo_enabled": false,
    "id": 483168465,
    "id_str": "483168465",
    "is_translator": false,
    "lang": "en",
    "listed_count": 7,
    "location": null,
    "name": "Justin",
    "notifications": null,
    "profile_background_color": "C0DEED",
    "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_tile": false,
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/483168465/1447963946",
    "profile_image_url": "http://pbs.twimg.com/profile_images/847917527183314946/lJHWtTdk_normal.jpg",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/847917527183314946/lJHWtTdk_normal.jpg",
    "profile_link_color": "1DA1F2",
    "profile_sidebar_border_color": "C0DEED",
    "profile_sidebar_fill_color": "DDEEF6",
    "profile_text_color": "333333",
    "profile_use_background_image": true,
    "protected": false,
    "screen_name": "Jut001",
    "statuses_count": 686,
    "time_zone": null,
    "translator_type": "none",
    "url": null,
    "utc_offset": null,
    "verified": false
  }
}

 

Nice work!! But now what do you do with that data? The tweets shown in the Logger block are not being sent anywhere other than the logger panel, and therefore the data cannot be accessed from anywhere else. It's not very useful to have it terminate into a Logger block. In the next step, you will publish the data.


Add and configure Publisher blocks

Publisher blocks allow you to send signals to a particular Topic. A topic is useful because it allows multiple services to share signals via the topic name.

Now that you have added a cumulative_count attribute to each of the signals coming from the Twitter streaming API, let's publish the signals to a topic.

  1. Drag a Publisher block onto your canvas.
  2. Configure the block as follows:
    1. Name: Publish Tesla Tweets
    2. Topic: workshop.tweets-count.tesla
  3. Click accept.
  4. Drag another Publisher block onto your canvas.
  5. Configure the block as follows:
    1. Name: Publish Netflix Tweets
    2. Topic: workshop.tweets-count.netflix
  6. Click accept.
  7. Change the service configuration to match the image at right.
  8. Click save to save the service.

Restart service

Your changes to the service will not take effect until the service is restarted and pulls in the new changes.

  1. Click stop and then start to restart your service.

Your signal no longer terminates in the Logger block, it instead goes into a Publisher block. A Publisher block can send your signal to any other service that subscribes to that topic in your nio system.

[info] Keeping Track of Topics

Use a naming convention to keep track of your topics. Here the topics are named using dot notation and include the system name, the service name, and a descriptive identifier if necessary. This makes it easy to both find a particular publisher block and to know what data it is publishing.


Summary

You have learned how to use the Twitter streaming API to drive signals through a nio service. You used query phrases (like "Tesla" and "Netflix") to filter a firehose of tweets, count them, and send the signal with the cumulative count to a Publisher block. You will learn how to subscribe to these signals in the next tutorial.


Extra credit

Here are some ideas to improve or expand the services you just created.

  • Could you add SpaceX to the Get Tesla Twitter block in the service so it searches for both Tesla and SpaceX?
  • How could you add a third company into the service for a three-way race?
  • How could you change the service to use a single Twitter block? Hint: parse the tweet text with a RegExFilter block.

Getting help

We're always happy to help with any questions you might have about the nio Platform. View the troubleshooting guide, search the documentation, or post your questions in the forum. You can also contact live support by clicking the chat icon in the lower-right corner of the nio System Designer.

Troubleshooting tips related specifically to this tutorial are presented below.

Error Possible Cause Solution
Error 420, Enhance Your Calm You have reached your Twitter API limit. Stop the tweets-count service for a while and let your Rate Limit reset.

 

proceed to tutorial 3: merging streams »
 

results matching ""

    No results matching ""