my net house

WAHEGURU….!

Hampering TimeScale-DB over Cup of Tea

Let’s hamper TimeScale DB over Tea and find out how much Day it can save:

Hardware Requirements:

OS: Ubuntu 16.04

CPU/RAM – Third generation i3 / 10 GB

Hyper Threading: Enabled

What is TimeScale?

If this is your First Try with Postgress, make sure You know How to reset Password for Postgress.

Following code we are able to do all the CRUD operations using SQL Language only.


--creating a simpe table

CREATE TABLE conditions (
time TIMESTAMP NOT NULL,
location TEXT NOT NULL,
temperature DOUBLE PRECISION NULL,
humidity DOUBLE PRECISION NULL);

-- creating a hypertable. now it wil be partioned with location

SELECT create_hypertable('conditions','time','location',4);

INSERT INTO conditions(time, location, temperature, humidity)
VALUES (NOW(), 'office', 70.0, 50.0);

SELECT * FROM conditions ORDER BY time DESC LIMIT 100;

-- upda<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>ting data on Hyper tables

UPDATE conditions SET temperature = 20000, humidity = 50000
WHERE time = '2019-02-17 12:45:02.189422' AND location = 'office';

SELECT * FROM conditions ORDER BY temperature DESC LIMIT 100;

-- upserting data

-- create it with unique IDs

CREATE TABLE conditions_unique (
time TIMESTAMPTZ NOT NULL,
location TEXT NOT NULL,
temperature DOUBLE PRECISION NULL,
humidity DOUBLE PRECISION NULL,
UNIQUE (time, location)
);
INSERT INTO conditions_unique(time, location, temperature, humidity)
VALUES (NOW(), 'office', 70.0, 50.0);
INSERT INTO conditions_unique
VALUES ('2019-02-17 12:56:05.080302+05:30', 'office', 898900000000.1, 9879678000000000.0)
ON CONFLICT DO NOTHING;

SELECT * FROM conditions_unique ORDER BY temperature DESC LIMIT 100;

It is as easy to play with bit complex quiries as one is doing Python. ūüėČ


-- Before going with this you have to download and setup NYC data
-- Folow this link https://docs.timescale.com/v1.2/tutorials/tutorial-hello-nyc

\c nyc_data

SELECT date_trunc('day', pickup_datetime) as day, avg(fare_amount)
FROM rides
WHERE passenger_count > 1 AND pickup_datetime < '2016-01-08'
GROUP BY day ORDER BY day;

SELECT date_trunc('day', pickup_datetime) as day, COUNT(*) FROM rides
GROUP BY day ORDER BY day
LIMIT 5;

SELECT time_bucket('5 minute', pickup_datetime) AS five_min, count(*)
FROM rides
WHERE pickup_datetime < '2016-01-01 02:00'
GROUP BY five_min ORDER BY five_min;

SELECT rate_code, COUNT(vendor_id) as num_trips FROM rides
WHERE pickup_datetime < '2016-01-08'
GROUP BY rate_code ORDER BY rate_code;

-- Join rides with rates to get more information on rate_code
SELECT rates.description, COUNT(vendor_id) as num_trips FROM rides
JOIN rates on rides.rate_code = rates.rate_code
WHERE pickup_datetime < '2016-01-08'
GROUP BY rates.description ORDER BY rates.description;

now let's Play with more complex stuff but see how far we can go?

  1. How large should I configure my intervals for time partitioning?
  2. Should I use space partitioning, and how many space partitions should I use?

Look for Best Practices:> https://docs.timescale.com/v1.2/using-timescaledb/hypertables


-- ============================================================
-- Structure
-- ============================================================

CREATE DATABASE test;

\c test;

CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;

CREATE TABLE collfact (
time TIMESTAMP NOT NULL,
pointid BIGINT NOT NULL,
value DOUBLE PRECISION NULL,
mesureid BIGINT NULL,
version INT NULL,
insertiondate TIMESTAMP NULL,
iscurrent CHAR(1)
);

SELECT create_hypertable('collfact', 'time', 'pointid', 16, chunk_time_interval => interval '1 day');
create index on collfact(time, pointid);

-- ============================================================
-- Generate test data, ~ 1 year
-- ============================================================

-- meter location, or point
create table point(
pointid BIGINT NOT NULL PRIMARY KEY,
pointnm VARCHAR(20) NOT NULL
);

CREATE SEQUENCE pointid_seq START 1;

CREATE OR REPLACE FUNCTION genPoints(in a_no_of_records integer) RETURNS integer AS $$
DECLARE
v_counter integer;
vhash bigint;
BEGIN
v_counter := 1;
RAISE NOTICE 'No of records to insert : %', a_no_of_records;
WHILE (v_counter <= a_no_of_records) LOOP
IF( v_counter % 10000 =0) THEN
RAISE NOTICE 'Counter here is %', v_counter;
END IF;
vhash := nextval('pointid_seq');

--insert into partiton table
INSERT INTO point(pointid, pointnm)
VALUES (vhash, 'Point num.' || vhash);
v_counter := v_counter + 1;
END LOOP;
RETURN 0;
END;

$$ LANGUAGE plpgsql;

-- Meter creation
select * from genPoints(10000);

create table time (
time TIMESTAMP NOT NULL PRIMARY KEY
);

-- Year 2018, step 15'
INSERT INTO time(time)
SELECT dd
FROM generate_series
(timestamp '2018-01-01 00:00:00'
, timestamp '2018-12-31 23:45:00'
, '15 minutes'::interval) dd ;

-- mesure type table; just 4 here
CREATE TABLE mesure (
mesureid BIGINT NOT NULL PRIMARY KEY,
mesurenm VARCHAR(20)
);

INSERT INTO MESURE (mesureid, mesurenm) VALUES (1, 'P+');
INSERT INTO MESURE (mesureid, mesurenm) VALUES (2, 'P-');
INSERT INTO MESURE (mesureid, mesurenm) VALUES (3, 'Q+');
INSERT INTO MESURE (mesureid, mesurenm) VALUES (4, 'Q-');

\timing on

-- INSERT for 1000 meters, 2 mesure types
insert into collfact (time, pointid, value, mesureid, version, insertiondate, iscurrent)
select time, pointid, trunc(random() * 99999), 1 as mesureid, 1, now(), '1'
from
(select * from time) as a,
(select * from point limit 1000) as b
UNION ALL
select time, pointid, trunc(random() * 99999), 2 as mesureid, 1, now(), '1'
from
(select * from time) as a,
(select * from point limit 1000) as b;

-- INSERT for 1000 meters, 2 mesure types
insert into collfact (time, pointid, value, mesureid, version, insertiondate, iscurrent)
select time, pointid, trunc(random() * 99999), 3 as mesureid, 1, now(), '1'
from
(select * from time) as a,
(select * from point limit 1000) as b;

I am running out of Shared memory after 15-20 mintues by using above code.

Yo can use code from this Git repo as well:

https://github.com/arshpreetsingh/timeScale-analysis

My findings:

1. You need to spend time over optmization of DB Setup like how much memory you want to share and should look for how to Optimize PGsql on your System.

2. You must carefully understand your data and how much Chunck-Size you want to define.

3. You don’t have to think much while writing complex Quiries because writing complex quiries is straight forward as well.

4. There is very good CRUD support.

5. You can perform Data-Analysis, Visualization and Machine-Learning very easily because community is supporting it on the FLY.

6. As it is extension of PG-SQL so stability can be defined.

Some Gitty GITs

When you don’t want to type username an password each time or when you are not allowed to do so as well.

Your repo should look like this:

git@github.com:USERNAME/REPOSITORY.git

you can see config using:


nano .git/config

SSH key can be generated like:

ssh-keygen -t rsa -b 4096 -C “arshpreet.singh@myemail.com”

 

More about Git and adding ssh key can be found here:
https://help.github.com/articles/connecting-to-github-with-ssh/

Click – Command Line Interface Creation Kit , OH yeahh Python

This is an Easy to do things in python so we can run scripts using command line Arguments for us as well.


import click 

@click.command()
@click.option('--count',default = 1, )
@click.option('--name',prompt = "What you are supposed to do here?")
def test_click(count,name):
    for i in range(count):
        click.echo(str(name)) 
'''
How to run this script:

python test_click.py --name "Arsh" --count 4

'''

Learn Ruby in Next 30 Minutes

Right Now time is: 5:00 , Let’s complete this by 5:30

# This is all about ROR on print statements :

puts ‘Hello, World!’
puts ‘Hello world’+’how are you’

# how to format a string in ruby

puts “The number is %d” % [20]

# This is al about Variables and Multiline comments:

=begin
in order,
a string,
a boolean,
and a number
=end
aString = “I’m a string!”
aBoolean = true
aNumber = 42
afloatMaybe = 10.0
afraction_could_be = 25/5
puts “this is a flaot %d” % [afraction_could_be]

# This is all about “UGLY” Math we are doing with Ruby ūüėõ muhahahhaha

numberOne = 6
numberTwo = 8
numberThree = 5
numberFour = 12
numberFive = 36

testOne = numberTwo*numberThree #fill in the blank to make this 40
testTwo = numberFive-6*numberOne #fill in the blank to make this 0
testThree = numberFour-numberThree #fill in the blank to make this 7
if testOne == 40 && testTwo == 0 && testThree == 7
puts true
end

# Lots about string Manipulation and stuff as well.

myString = “Hi! I am code!” #In the next line, use methods to change it.
myNewString = [myString.upcase,myString.downcase,myString.reverse]
puts myNewString

# so much more about Arrays and processing:

myArray = [“Not me!”, “Not me!”, “Me!”, “Not me!”]
arr = [11,22,33,44,55]
arr.each_with_index {|val, index| puts “#{index >= 0 && index “this is my hash value 1″,”Mum”=>”this is my Hash value2”}
puts myFirstHash[“Dad”]

# This condition is all about If ELse and bla bla bla

language = “No Language”
if language === “English”
puts “Hello!”
elsif language === “Spanish”
puts “Hola!”
else
puts “this is the condition where all is well”
end

# unless, until and bla bla bla
#Make your loop below!
a = 2
b = 4
unless a>b
puts “omg!!!”
end

until a>b
puts “oye-hoyeeee”
end

# this is all about methods we need to know:

def methodname
puts “how to execute methods”
end

methodname()

For classes start from here:
https://www.tutorialspoint.com/ruby/ruby_classes.htm

OOPS it is still on. ūüė¶

Python Send SMS all over the things

This is Python Script that you can use to send Messages on any Phone without leaving trace of any number.


# python script for sending message update 

import time 
from time import sleep 
from sinchsms import SinchSMS 

# function for sending SMS 
def sendSMS(): 

	# enter all the details 
	# get app_key and app_secret by registering 
	# a app on sinchSMS 
	number = '+919915959387'
	app_key = '29d0e101-e619-4226-823a-c39c4b99b5ac'
	app_secret = '1WkUjJ3RWEy1P9h9mw/2sQ=='

	# enter the message to be sent 
	message = "hello"

	client = SinchSMS(app_key, app_secret) 
	print("Sending '%s' to %s" % (message, number)) 

	response = client.send_message(number, message) 
	message_id = response['messageId'] 
	response = client.check_status(message_id) 

	# keep trying unless the status retured is Successful 
	while response['status'] != 'Successful': 
		print(response['status']) 
		time.sleep(1) 
		response = client.check_status(message_id) 

	print(response['status']) 

if __name__ == "__main__": 
	sendSMS()
 

Logging and importance of it with examples

Python and Flask supports wide range of logging as well. Either it’s warning, error or just a logger you can ago through all of those in very specific instance of time.


Logging is important of the Maintainability of the application.

Now logging is something like you need to go for when you see or feel that your web app needs lots of “Watching as well!”

Here is simple Example in Flask:

import logging
from logging.handlers import RotatingFileHandler

from flask import Flask

app = Flask(__name__)

@app.route('/')
def foo():
app.logger.warning('A warning occurred (%d apples)', 42)
app.logger.error('An error occurred')
app.logger.info('Info')
return "foo"

if __name__ == '__main__':
handler = RotatingFileHandler('foo.log', maxBytes=10000, backupCount=1)
handler.setLevel(logging.INFO)
app.logger.addHandler(handler)
app.run()

[/code ]

for more detailed view on logging and system you can go for the following link as well. :

It’s very much explanatory: https://gist.github.com/mariocj89/73824162a3e35d50db8e758a42e39aab

install,setup and Play with Mongo Python

This is the good way to do stuff with Mongo, Python and other.

https://hevodata.com/blog/install-mongodb-on-ubuntu/

 

Deploy Micro-service in Seconds Using Falcon

Falcon framework is great when you have to build micro-services in Python, All we need to do is create a Class that would provide the specific Purpose to Establish one Micro-service as Route and add the route to the API would be able to server as deploy multiple modes inside the Falcon Architecture as well.


# sample.py

import falcon

class QuoteResource:
def on_get(self, req, resp):
"""Handles GET requests"""
quote = {
'quote': (
"I've always been more interested in "
"the future than in the past."
),
'author': 'Grace Hopper'
}

resp.media = quote
class HelpObject:
def on_get(self, req, resp):
"""Handles GET requests"""
quote = {
'quote': (
"I've always been more interested in "
"the future than in the past."
),
'author': 'Arshpreet Singh'
}

resp.media = quote

api = falcon.API()
api.add_route('/quote', QuoteResource())
api.add_route('/quote2', HelpObject())
<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;">&#65279;</span>

IBM hyper ledger Business Card creation and Code familiarity [inbetween]

model/*.cto file- This file is able to describe Asset, Participant and Transactions those occur at that business network as well .

  • Namespace
  • Resources
  • Imports from other namespaces, as required

If your model is very large, you can have multiple .cto model files, as necessary. Every .cto model file must include a single namespace and at least one resource definition.

NameSpace: NameSpace is just naming convention for the File-System. For each of your .cto file namespace would be an unique name for the file. Each of the resources available in the file would be taken from that namespace.

Resource: Now we have following kind of resources we can use.

Asset: A business network Asset(Example Bit-coin is an Asset in Bit-coin-Network)

Transaction: A business Logic

Participant: A business network Participant

Event: Notification of something happening in the network

Enumerated Type: A set of named Values

Concept: Any object you want to model that is not one of the other types

Each resource type corresponds to its model type of the same name (for example, asset is used to model an Asset, participant models a Participant, etc.)

 

asset Vehicle identified by vehicle_id {

 

 

}

Disruption of Block-Chain in Present and Future

An immutable and distributed ledger with unique set of transactions in the forms of blocks depends on the nature and cohesiveness of each transaction which is either able to form the solution or disrupt the already existing solutions is the main event for the current existence and hype for the Block-Chain and it’s nature in the present and future.

There are certain core features of Block-Chain which includes transparency,  democracy,  Efficiency, decentralization and security, Such features can also be considered as default ones because It is the real nature of Block-Chain the provides such basic norms on the FLY!

  1. Banking: obviously!

If we talk about how and when Banking is going to be into Block-chain and how much that could bring out real change into the things then things are already there to have fun with!

Why we need Banks? – A basic institution that is able to save your money like a wallet and you are able to send it all over the world. Right now just believe that we are not talking about Investments as well.

Download ABRA and just Enjoy the Bank in an APP: https://www.abra.com/

Barclyas is the bank that uses the BlockChain:

15% of Banks will be using Block-Chain by the end of 2017:

2. Cyber Security:

Data is stored in the encrypted form and each time it is verified by multiple parties while storing it in DB so that means each time Data is able to secure by default. That eliminates the need of Third party security systems!

3.  Supply-Chain Management! (OMG that was DOPE when I heard that how Efficient it could be in Supply-Chain management):

All the transactions are Documented in permanent record and are being watched securly when any assest is moved from one party to another in the supply chain management!  That is able to measure time, Cost, labor and Emissions which is helpful to reduce and form the necesary  plan to make complete Efficient Supply-Chian management! Which is also helpful to verifity the authenticity and fair trade status of the product.

Startups that are working on Supply-Chain Management are as follows:

https://www.provenance.org/whitepaper

https://bravenewcoin.com/news/fluent-unveils-blockchain-based-global-supply-chain-software-platform/

http://www.skuchain.com/

http://www.blockverify.io/

4. Forecasting:

Think of world like where each and every person is able to collaborate their brian-power and able to build and things collaboratly either it is a research work or making a staretgy that is able to predict next stock price in the NASDAQ!

A decentralized prediction market solution: http://www.augur.net/

5. Networking and IOT:

Thinks of different nodes and how these are able to communicate with each other that could bring repairing and updating and repairing itself without even need of external human maintainer! All the nodes would be able to connecet to each other with eliminating the need of centralized system.

One can see whole idea about ADEPT here: https://www.slideshare.net/_hd/ibm-adept

6. Insurance:

Insurance is based on trust management. BlocakChain can be used to verify many types of data in insurance contracts! Like the real identity of insured person as well as his/her medical records as well.

One example of such system is Aeternity: https://aeternity.com/

7.  Private Transport and Ride Sharing:

There are various cases of Block-Chain which can  be helpful for peer to peer ride sharing system. Presently most of the Ride sharing or Taxi booking systems are based on the some kind of Centralized system like Uber and OLA but using Peer to Peer Transactions and communication can be improved and cost of middle man or centralized system can be reduced over a large amount!

Great Examples are Arcade City and Lazooz:

https://cointelegraph.com/news/arcade-city-decentralized-blockchain-based-answer-to-uber

http://lazooz.org/

8.   Cloud Storage:

It was always a challenge  only because we always need to secure data and have it in many different places and those places should be on virtual locations not in one but also in many!

Storaj is one example of that: https://storj.io/

9. Charity:

There are many cases of corruption comes into place when there are money collection and resources utilization in the charity, There is no doubt that in the cases of Charity because of less transparency all the Money of Charity could be involved into wrong things as well. BlcokChain could be a very useful to get to know that where the donations are going!

https://www.bitgivefoundation.org/

 

10.  Voting:

Block-chain can also be used for Registration, Verification and Vote-counting. Creating and immutable, Publicly viewable ledger of records is the real way to make elections more democratic and viewable.

https://www.democracy.earth/

https://followmyvote.com/

11. Government:  Most of the government based systems are slow, in-efficient and biased. To slow down the bureaucracy,increase security, efficiency and to make government systems more transparent BlockChain is so far the best solution as well.

https://www.khaleejtimes.com/editorials-columns/blockchain-future-is-here-and-dubai-has-grabbed-it

12. Public Benefits:

Public Benefits is the one of the system that has suffered each time from slowness and bureaucracy, As one of the major thing of being transparent block-chain here is the great player to implement that all benefits towards public can be shared in transparent way.

Govcoin is one of the example of BlockChain that is disrupting welfare  system in the UK:

https://www.blockchaintechnology-news.com/2017/11/28/govcoin-aims-disrupt-uk-welfare-system/

There is one more project which is called circle that is able to implement Universal Basic Income using BlockChain: https://www.circle.com/in/

13: Health-Care:

Hospitals always  lack of need to save secure data and that can be used foe various treatment in the patients. BlockChain can be used to save and secure Medical data of patients and that can be shared with Authorized doctors and patients.

Following are the two startups working on the BlockChain to work on secure data:

https://gem.co/

https://tierion.com

14.  Energy Management: Energy Producers and users  have to go through a long list of middle grids to get resources from each other.

Transactive Grid is blockchain solution that is able to create De-Centralized way share resources between various energy producers and consumers.  Transactive Grid uses the Ethereum Blockchain.

15. Online Music:

There are several Block-Chain solutions those are working on the problem for selling music from producers to users. Smart contracts can also be used directly to solve the issue of licensing.

http://myceliaformusic.org/

https://ujomusic.com/

16. Retail:

Shopping is always based on trust on the shop or the market place but in real there is no particular way to understand that how good or bad product could be inside the wrapper. BlockChain is able to cut all the middle-man between buyers and sellers so each time time. Smart contracts are there to maintain the credibility of the system.

https://www.openbazaar.org/

https://ob1.io/

17. Real State: There are few issues in real estate like Fraud, Lack of transparency and  mistaken in public records. BlockChain can simply reduce the paper based record keeping structure. Block-Chain implementation can simply help with tracking, verifying ownership, ensuring accuracy of documents and transferring the deeds.

The one company that is solving this problem is https://www.ubitquity.io/web/index.html

18. Crowd-Funding:

In Blcok-Chain Crowd-funding is created through smart-contarcts and online reputation system. New products can raise funds by releasing their own tokens and those tokens can be used later to buy products and other services.

19 OUR industry: Telecommunication and Networking. SDN(Software Defined Networking!)

Securechain delivers security, scalability and auditability to Software Defined Networks.

SYTEL REPLY HAS DEVELOPED A WORKING PROOF-OF-CONCEPT TO SHOW THE POTENTIAL OF THIS SYSTEM, BASED ON THE ETHEREUM BLOCKCHAIN (THE SAME BLOCKCHAIN AS CHOSEN BY THE R3 FINANCIAL CONSORTIUM, MICROSOFT
AND IBM/SAMSUNG), WHICH CAN NOT ONLY ALLOW VALID NETWORK ENTITIES SUCH AS SWITCHES TO ENTER THE NETWORK, BUT TO DETECT AND REJECT HACKING ATTEMPTS.
Paper based on Block-Chain and SDN:
%d bloggers like this: