Simple Example of File processing using Spring Integration
Here is my use case , watch directory for new file and process files of certain format ( e.g *.txt ) . Process them and send to Message queue .
Spring integration can be used for both file processing and message queue . This blog provides example for watching file folder for changes and invoking a message
Its elegant and straight forward , example here watches directory in a certain directory and then process the files of type text .
Note : Example here will get you started , if you need to implement in load balanced environment , the "AcceptOnceFileListFilter" may not be enough , you will need persistent storage and use "FileSystemPersistentAcceptOnceFileListFilter"
First project that I wanted to try when I looked at GrovePi and the connectors .
Architecture has 3 main components
Edge - Get data temperature , light and humid readings and publish to the server
Data Platform - Consume the data provided by Edge and store . Provide api to get analysis on the data
Client - this can be anything that consumes rest api , mobile app or Alexa ( its fun this way !)
Edge
Raspberry Pi and GrovePi module with the connectors act as the Edge .
Python module continuously reads the readings every minute and pushes
to the server using Rest api .
API interfaces to consume the data from edge and store . Developed using Spring Boot
Data can be stored in any db of your choice based on the analytical needs .
This project I have used Redis to store maximum of
100 values to provide simple analytics ( Average and Latest readings) on that data .
Readings are also exposed as rest api .
It's a fun project and just to get hands on IOT . Edge , Platform and Client
are totally decoupled and can be easily changed for a different use case .
Although it's a simple project it get exposure to variety of technologies
Python , Java , Spring Boot , Redis , Node js and AWS Lambda's
Demo
Did you find it useful , did you try it , hack it , improve it ?
do share in the comments below
Started playing with the GrovePi Starter kit for RaspberryPi.
I chose GrovePi this as there is no complex wiring, have a look at how nicely it fits the Raspberry pi .
After that I did couple of others , like home weather display and twitter feed .
I got excited now I want to do more , I want to do analytics . So was thinking about gathering data from sensors- temperature , humidity , sound and light readings and send it to thecloud for doing analytics . Python was new to me , but want to use it any way as I can learn a new language.It was easy and I wasn't stuck with any thing .It was fun learning as well . Go Python !
Data Layer
Now I have to push data to a Data layer . Why not use an instance in Google Cloud (they are giving 300$ in free credit!) .Rest API in cloud will be based on Java and Spring Boot .
Data will be stored in redis , as I don't want to store a lot of data . Just last 100 readings
or so and get the current and average readings from the sensors ( this project info is in http://ajaxrocks.blogspot.com/2017/07/fun-with-iot.html ) for this but current integration is only for PubNub
Alexa Skill
While all this was going on , I see Alexa app demos . So I started looking into that
as well . Started building a sample app , using AWS Lambda's .. hey!! ..always want to get my hands on server-less programming as well . Alexa sample app Tutorial used node js .. this is good , I can sharpen my node js skills . Okay I built the sample app .. It was so much fun , the sample app used Facts as an array (that are hardcoded ) . When you invoke it using Alexa , it chooses one of the Fact
randomly . Fun part is you change all the facts and then keep asking facts !! . See programming can be fun
Picture Time ..
Concept is actually the time triggered to take
picture instead of LED blink
Messaging using PubNub
While all this was going on , I learnt about PubNub in DevNetCreate conference
It provides Infrastructure as a Service for doing messaging . they have SDK for almost all the languages I use .Cool , now I can invoke the GroveSensors ( Light on ! ) using PubNub message . Tried it it worked great .. hey .. Next part if you haven't already guessed I can use Alexa to invoke it .. Yeah !
Rewiring to use API
All set now I invoke , node js call to PubNub worked great .Deployed to AWS , now it just doesn't want to work !! Hmmmm... may be I am doing something wrong .. setup security , network interfaces .. still not working .. Don't know what the real reason was .Why waste time on this , let me set this up as a rest api in google cloud and use Alexa skill ( node js) to call that api... hey that can work .... setup the api .. changed the alexa app and everything worked like a charm
Shoutout to Alexa SDK, Google Cloud , PubNub , Raspberry Pi and Grove Pi ... Thanks Guys
Architecture
Learn it ,Use it , Hack it
This project was fun , it also helped understand technologies involved with IOT better .
Learn new languages in a very pragmatic way . I have all the code I used in the github links
, so play with them , hack them for your use case , make them better ..
