Easy Analytics from Web-Sourced Data

Easy Analytics from Web-Sourced Data

Easy Analytics from Web-Sourced Data

April 7, 2015 / Posted By: wotio team

Let’s create a Bip that Tracks some Stock Prices and saves it for us out to an Analytics Service. We'll use keen.io for this example, but you could just as easily use mongodb or other pods and run the data through whatever services you want as well.

This data could be anything that’s found on the web. So although this example is polling for a current stock price, any data that can be scraped from the web is fair game. We’ll show how that’s done in a bit.

First lets define which stock symbol we’d like to track. Big Blue is as good as any, so lets pick the (1) Flow Controls and (2) ‘Generate a Payload' with (3) “IBM" as the payload.




This tells our bip to start out emitting "IBM" into whatever we want to connect to it.
We'll want to bind that "IBM" payload to a specific HTML DOM element on a specific page, so let's go ahead and use the HTML pod DOM Selector action (4) to make a web request and use the jQuery Selector to get the data we want. We'll add that action the same way we added our initial payload. (Create New Action -> HTML pod -> DOM selector)

(5) Go ahead and add our new DOM Selector action onto the graph (5) and double-click to edit the properties. (6)
(6) Here we’re going to go out to the web to grab the latest Stock Price. We'll use Google Finance for that as you can see in the URL field. We're interested in a specific DOM element, so we're going to enter #price-panel .pr span to select the page element we're interested in (the price!)

Do you see how we used the generated payload of "IBM" in the first step to build out our URL query in this step? That's how we grab a piece of dynamic data from one pod and feed it into another in whichever way we want!

Now that we're grabbing some data (IBM listed price), we need a place to put it. Well, that's exactly what Keen.io was built for, so let's use the keen.io pod to send our data over to our keen.io account!

Again, adding a Keen.io action to our bip is done the same way we added our Payload Generator & DOM Selector actions. (Pods -> Keen.io -> Add an Event)

Once we've added Keen.io, go ahead an add the action to the graph and its simply a matter of connecting the dots!

By connecting the actions in this way, as if by magic we now have available to us the transformed or collected attributes from the other actions.

Keen.io expects to recieve a well formed key:value object, so we'll marshal the Event Object field to send over a JSON object keyed on "IBM" with the value dynamically set to the Text field from our DOM Selector pod action.

With those things set, let's run our bip and then head over to our keen.io dashboard to see that everything is flowing correctly.

Viewing the latest data event we sent to keen.io (via our bip!) and sure enough we see that our stock price value is getting sent correctly.

{ "keen": { "timestamp": "2015-04-06T14:00:01.257Z", "created_at": "2015-04-06T14:00:01.257Z", "id": "5522916196773d1d96613471" }, "IBM": "159.08" }


Now we can take advantage of all of the rich analytics and monitoring that Keen.io provides on that data we've sent.

To recap:
1. Pods -> Create Event -> Flow -> Generate Payload
2. My Bips -> Create a Bip -> Add Generate Payload
3. Add an Event -> HTML -> DOM Selector
4. Add an Event -> Keen.io -> Add an Event
5. Click-n-drag to connect Payload -to- DOM-Selector
5a. Click-n-drag to connect DOM-Selector -to- Keenio-Event
6. Double Click Each Pod Icon To Configure the Actions.
7. Hit Run!

Feel free to follow these steps to build up a tracker for your own portfolio.

And of course with bip.io we can connect many different services to store and mix-n-match whatever data we want, coming from whatever website or web-service we want, pretty easily.

So get out there and start bipping!