my net house

WAHEGURU….!

Category Archives: Ubuntu

Install Vanilla Kubernetes Cluster on Ubuntu.

Here We are Installing Vanilla Flavour of Kubernetes Cluster(With High Availablility), Which is Small but Production-Ready.

 

Prerequisites:

 

  • You need to have at least Two VMs(Master Node and Slave Node)
  • Both VMs should connected on the Same Network.
  • Master Node must have 2 CPUs and at least 2 GB of RAM.
  • Swap must be turned OFF on Both of the VM(s).
  • A basic understanding of Linux, Networking and docker unless you are a magician. 😉

 

Installation Steps:

Run Following Command on Master Node.(Without Hash 😛 ) 
#  apt install docker.io

# systemctl enable docker

# apt install curl

# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add

# apt-add-repository “deb http://apt.kubernetes.io/ kubernetes-xenial main”

# apt install kubeadm

After Running all the Commands above you would be able to Install All Kubernetes Packages on the Master Node.

Now you need to Deploy Cluster. Run Following Commands. 

 

# swapoff -a

#  hostnamectl set-hostname master-node

# kubeadm init –pod-network-cidr=10.244.0.0/16 (Look at it’s output Carefully!!)

# mkdir -p $HOME/.kube

# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

# chown $(id -u):$(id -g) $HOME/.kube/config

 

 

Deploy Pod Network: (There are different Types of Networks you can use. Here we are using flannel)

 

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

 

Add the slave node to the network in order to form a cluster. 

Run the Command which you will get in output after running this:
kubeadm init –pod-network-cidr=10.244.0.0/16” on you Master Node. make sure you have to run Command which will lokk like Following on Your Slave-node. So you can use Kubernetes Cluster. 

** Don’t Run Following Command as it is.

 

kubeadm join 172.31.9.252:6443 –token bixhtx.4bqyhe45g8fvhqeq \

    –discovery-token-ca-cert-hash sha256:eb5f21bfda175f8d60ca50780941df83169893182c8fd2073de32daccc7b2e6d

 

Other Comman Kubernetes commands to Play with your Cluster:

# kubectl get nodes

# kubectl get pods –all-namespaces

 

Enjoy your Cluster!! 

TPOT Python Example to Build Pipeline for AAPL

This is  just first Quick and Fast Post.

TPOT Research  Paper: https://arxiv.org/pdf/1702.01780.pdf


import datetime
import numpy as np
import pandas as pd
import sklearn
from pandas_datareader import data as read_data
from tpot import TPOTClassifier
from sklearn.model_selection import train_test_split

apple_data = read_data.get_data_yahoo("AAPL")
df = pd.DataFrame(index=apple_data.index)
df['price']=apple_data.Open
df['daily_returns']=df['price'].pct_change().fillna(0.0001)
df['multiple_day_returns'] =  df['price'].pct_change(3)
df['rolling_mean'] = df['daily_returns'].rolling(window = 4,center=False).mean()

df['time_lagged'] = df['price']-df['price'].shift(-2)

df['direction'] = np.sign(df['daily_returns'])
Y = df['direction']
X=df[['price','daily_returns','multiple_day_returns','rolling_mean']].fillna(0.0001)

X_train, X_test, y_train, y_test = train_test_split(X,Y,train_size=0.75, test_size=0.25)

tpot = TPOTClassifier(generations=50, population_size=50, verbosity=2)
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
tpot.export('tpot_aapl_pipeline.py')

The Python file It returned: Which is real Code one can use to Create Trading Strategy. TPOT helped to Selected Algorithms and Value of It’s features. right now we have only provided ‘price’,’daily_returns’,’multiple_day_returns’,’rolling_mean’ to predict Target. One can use multiple features and implement as per the requirement.


import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

# NOTE: Make sure that the class is labeled 'target' in the data file
tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)
features = tpot_data.drop('target', axis=1).values
training_features, testing_features, training_target, testing_target = \
            train_test_split(features, tpot_data['target'].values, random_state=42)

# Score on the training set was:1.0
exported_pipeline = GradientBoostingClassifier(learning_rate=0.5, max_depth=7, max_features=0.7500000000000001, min_samples_leaf=11, min_samples_split=12, n_estimators=100, subsample=0.7500000000000001)

exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)

Python for text processing

Python is more about ‘Programming like Hacker’ while writing your code if you keep things in mind like reference counting, type-checking, data manipulation, using stacks, managing variables, eliminating usage of lists, using less and less “for” loops could really warm up your code for great looking code as well as less usage of CPU-resources with great Speed.

Slower than C:

Yes Python is slower than C but you really need to ask yourself that what is fast or what you really want to do. There are several methods to write Fibonacci in Python. Most popular is one using ‘for loop’ only because most of the programmers coming from C background uses lots and lots of for loops for iteration. Python has for loops as well but if you really can avoid for loop by using internal-loops provided by Python Data Structures and Numpy like libraries for array handling You will have Win-Win situation most of the times. 🙂

Now let’s go with some Python tricks those are Super cool if you are the one who manipulates,Filter,Extract,parse data most of the time in your job.

Python has many inbuilt methods text processing methods:

>>> m = ['i am amazing in all the ways I should have']

>>> m[0]

'i am amazing in all the ways I should have'

>>> m[0].split()

['i', 'am', 'amazing', 'in', 'all', 'the', 'ways', 'I', 'should', 'have']

>>> n = m[0].split()

>>> n[2:]

['amazing', 'in', 'all', 'the', 'ways', 'I', 'should', 'have']

>>> n[0:2]

['i', 'am']

>>> n[-2]

'should'

>>>

>>> n[:-2]

['i', 'am', 'amazing', 'in', 'all', 'the', 'ways', 'I']

>>> n[::-2]

['have', 'I', 'the', 'in', 'am']

Those are uses of lists to do string manipulation. Yeah no for loops.

Interesting portions of Collections module:

Now let’s talk about collections.

Counter is just my personal favorite.

When you have to go through ‘BIG’ lists and see what are actually occurrences:

from collections import Counter

>>> Counter(xrange(10))

Counter({0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1, 8: 1, 9: 1})

>>> just_list_again = Counter(xrange(10))

>>> just_list_again_is_dict = just_list_again

>>> just_list_again_is_dict[1]

1

>>> just_list_again_is_dict[2]

1

>>> just_list_again_is_dict[3]

1

>>> just_list_again_is_dict['3']

0

Some other methods using counter:

Counter('abraakadabraaaaa')

Counter({'a': 10, 'r': 2, 'b': 2, 'k': 1, 'd': 1})

>>> c1=Counter('abraakadabraaaaa')

>>> c1.most_common(4)

[('a', 10), ('r', 2), ('b', 2), ('k', 1)]

>>> c1['b']

2

>>> c1['b'] # work as dictionary

2

>>> c1['k'] # work as dictionary

1

>>> type(c1)

<class 'collections.Counter'>

>>> c1['b'] = 20

>>> c1.most_common(4)

[('b', 20), ('a', 10), ('r', 2), ('k', 1)]

>>> c1['b'] += 20

>>> c1.most_common(4)

[('b', 40), ('a', 10), ('r', 2), ('k', 1)]

>>> c1.most_common(4)

[('b', 20), ('a', 10), ('r', 2), ('k', 1)]

Aithematic and uniary operations:

>>> from collections import Counter

>>> c1=Counter('hello hihi hoo')

>>> +c1

Counter({'h': 4, 'o': 3, ' ': 2, 'i': 2, 'l': 2, 'e': 1})

>>> -c1

Counter()

>>> c1['x']

0

Counter is like a dictionary but it also considers the counting important of all the content you are looking for. So you can plot the stuff on Graphs.

OrderedDict:

it makes your chunks of data into meaningful manner.

>>> from collections import OrderedDict
>>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2}
>>> new_d = OrderedDict(sorted(d.items()))
>>> new_d
OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)])
>>> for key in new_d:
...     print (key, new_d[key])
... 
apple 4
banana 3
orange 2
pear 1

Namedtuple:

Think it the way you need to save each line of your CSV into list of lines but along with that you also need to take care of not just the memory but as well as You should be able to store each line as dictionary data structure so if you are fetching lines from Excel or CSV document which comes in place when you work at Data-Processing environment.

# The primitive approach
lat_lng = (37.78, -122.40)
print 'The latitude is %f' % lat_lng[0]
print 'The longitude is %f' % lat_lng[1]

# The glorious namedtuple
LatLng = namedtuple('LatLng', ['latitude', 'longitude'])
lat_lng = LatLng(37.78, -122.40)
print 'The latitude is %f' % lat_lng.latitude
print 'The longitude is %f' % lat_lng.longitude

ChainMap:

It is Container of Containers: Yes that’s really true. 🙂

You better be above Python3.3 to try this code.

>>> from collections import ChainMap

>>> a1 = {'m':2,'n':20,'r':490}

>>> a2 = {'m':34,'n':32,'z':90}

>>> chain = ChainMap(a1,a2)

>>> chain

ChainMap({'n': 20, 'm': 2, 'r': 490}, {'n': 32, 'm': 34, 'z': 90})

>>> chain['n']

20

# let me make sure one thing, It does not combines the dictionaries instead chain them.

>>> new_chain = ChainMap({'a':22,'n':27},chain)

>>> new_chain['a']

22

>>> new_chain['n']

27

Comprehensions:

You can also do comprehensions with dictionaries or sets as well.

>>> m = {'a': 1, 'b': 2, 'c': 3, 'd': 4}

>>> m

{'d': 4, 'a': 1, 'b': 2, 'c': 3}

>>> {v: k for k, v in m.items()}

{1: 'a', 2: 'b', 3: 'c', 4: 'd'}


StartsWith and EndsWith methods for String Processing:

Startswith, endswith. All things have a start and an end. Often we need to test the starts and ends of strings. We use the startswith and endswith methods.

phrase = "cat, dog and bird"

# See if the phrase starts with these strings.
if phrase.startswith("cat"):
    print(True)

if phrase.startswith("cat, dog"):
    print(True)

# It does not start with this string.
if not phrase.startswith("elephant"):
    print(False)

Output

True
True
False

Map and IMap as inbuilt functions for iteration:

map is rebuilt in Python3 using generators expressions under the hood which helps to save lot of memory but in Python2 map uses dictionary like expressions so you can use ‘itertools’ module in python2 and in itertools the name of map function is changed to imap.(from itertools import imap)

>>>m = lambda x:x*x
>>>print m
 at 0x7f61acf9a9b0>
>>>print m(3)
9

# now as we understand lamda returns the values of expressions for various functions as well, one just have to look
# for various other stuff when you really takes care of other things

>>>my_sequence = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
>>>print map(m,my_sequence)
[1,4,9,16,25,36,49,64,81,100,121,144,169,196,225,256,289,324,361,400]

#so square is applied on each element without using any loop or if.

For more on map,reduce and filter you can fetch following jupyter notebook from my Github:

http://github.com/arshpreetsingh

A simple script to do parsing of large file and save it to Numpy array

A normal approach:

huge_file = 'huge_file_location'
import re
import numpy as np
my_regex=re.compile(r'tt\d\d\d\d\d\d\d') #using a compiled regex saves the time
a=np.array([]) # just an array to save all the files

with open(file_location,'r') as f: # almost default method to open file
    m = re.findall(my_regex,f.read())
    np_array = np.append(a,m)

print np_array
print np_array.size
print 'unique'
print np.unique(np_array) # removing duplicate entries from array
print np.unique(np_array).size
np.save('BIG_ARRAY_LOCATION',np.unique(np_array))

In the above code f.read() saves big chuck of string into memory that is about 8GB in present situation. let’s fire up Generators.

A bit improved version:

def read_in_chunks(file_object):
   while True:
       data = file_object.read()
           if not data:
               break
        yield data

import numpy as np
import re
a=np.array([])
my_regex=re.compile(r'tt\d\d\d\d\d\d\d')
f = open(file_location)
for piece in read_in_chunks(f):
m = re.findall(my_regex,piece) # but still this is bottle neck
np_array = np.append(a,m)
print np_array
print np_array.size
print 'unique'
print np.unique(np_array)
print np.unique(np_array).size

A little bit faster code:

file_location = '/home/metal-machine/Desktop/nohup.out'
def read_in_chunks(file_object):
    while True:
        data = file_object.read()
           if not data:
               break
        yield data

import numpy as np
import re
a=np.array([])
my_regex=re.compile(r'tt\d\d\d\d\d\d\d')
f = open(file_location)

def iterate_regex():
''' trying to run iterator on matched list of strings as well'''
    for piece in read_in_chunks(f):
    yield re.findall(my_regex,piece)
for i in iterate_regex():
    np_array = np.append(a,i)
    print np_array
    print np_array.size
    print 'unique'
    print np.unique(np_array)
    print np.unique(np_array).size

But why performance is still not taht good? Hmmm…… How about Running it with Lots of Cython?

Look at the CPU usage running on Google instance 8Core system.

 
cpu-usage.png

Docker and other important things

One thing I am sure about now is there are toys around me whose I need to learn to play.. ❤

REST Python API

Requests module

Docker

Django

WSGI-> well what I know is it is medium between Webserver and your Application.

Application <WSGI> Python-application

Docker… Why?

It is Linux container, You can use containers to development.

Container seems better options to do things rather than normal..

Dear GOD, Please provide me chance to play with it!

Dpkg packages fix Bodhi/ubuntu/debian

$ sudo dpkg –configure -a
$ sudo apt-get –fix-missing
$ sudo apt-get -f install
$ sudo apt-get autoclean
$ sudo apt-get autoremove
$ sudo apt-get update
$ sudo apt-get upgrade
Bingooooo….!!

Bodhi linux , Bodhi and linux ,Bodhi and linux in india

THIS ARTICLE IS FICTIONS ON LINUX AND THE RISE OF BODHI LINUX, PLEASE DON’T TAKE IT SERIOUSLY AT ANY COST 😉

What is Linux:
http://en.wikipedia.org/wiki/Linux
http://www.linux.com/
http://www.linux.org/

There are millions or may be billions pages are available about Linux, Which you can find on the Google who also use Linux. Google has also promoted Linux very well to get great and stable services. Like Android which is also a fork of Linux. IMHO Linux is just not an operating system it is a thinking and when we use Linux we don’t use it we do Linux-ism. 😀

So that was all I wanted to say about Linux.
Now let us go to bodhi linux which is now one of my favorite Linux distro ever.
First: Who is Bodhi Linux
http://www.bodhilinux.com/
http://forums.bodhilinux.com/
http://www.bodhilinux.com/downloads_desktop.php

Bodhi Linux is the minimalist. Well it depends on how bodhi linux stuck in my mind. This is just the linux distro I ever wanted. That is Bodhi Linux. while tlking about Bodhi linux there are many other concepts those comes in our mind about Linux distros.

What actually all Linux user want from linux?

Light weight and fastest, Yes true. We want Light weight and fastest Linux distro because that is the actual standard of Linux but Linux with Gnome3,XFCE,ABC,XYZ, there are also good distros of Linux but not as good as Bodhi Linux. So we want fastest and Light weight linux.(Men’s style) But as we know there are lots of girls are taking lots of interests in the Linux so they want their Linux little bit dashy,sparky and styliey (I don’t know actual meaning of these words I just Listen from some of the girls :D). Due to this lots of graphical Linux developments has been started like Bodhi Linux. I am sure creator must be Indian because from the name of bodhi Linux we can see the Indian name Bodhi.

Basically Bodhi is Indian name:
http://en.wikipedia.org/wiki/Bodhi
http://www.facebook.com/wearebodhi
bodhi.in/

It is not Just enough that Bodhi is only a class of people or a clothes brand in bodhi, Bodhi is also a scared tree of India we also call it Bohad in Punjab. You can search the Images of Bodhi tree on Google by just typing the keywords “bodhi tree” or “bodhi india” or “india bodhi”(I just try to did some SEO trick for bodhi and linux here 😉 ) You can also see in the following Link.
http://en.wikipedia.org/wiki/Bodhi_Tree

Now you got how cool Bodhi and Linux are. Either the come together bodhi Linux or just separated bodhi and linux.

Now lets talk about features of bohdi linux.

Bodhi Linux is fastest Linux. When I say that about bodhi Linux you must have a question about Why it is fastest Linux? Number one about bodhi linux is that: Bodhi linux uses one of the light weight Linux desktop environment which is Enlightenment. You can find more about Bodhi Linux and enlightment in the following Linux related to Bodhi Linux and enlightenment.
https://www.enlightenment.org/
http://www.bodhilinux.com/e17guide/e17guideEN/
http://www.raspberrypi.org/forums/viewtopic.php?f=66&t=20423
Nuber two is ease of use of Enlightenment: Which is one of the biggest reason for bodhi linux to be one of the fastest Linux. It is common fact that how easy something will be to operate we will operate it fast. Just like hand pump is easy to operate than throwing a basket in the well. Using the handpump we can do much work with less time and less effort. (ahh handpump wali example tan siraa e c 😀 ) so using Bodhi linux due to ease of use we can do more work with bodhi in lesser time. Third point is smart design of bodhi and enlightenment. As I have already told you about hand pump in the bodhi Linux example, Ease of use comes from the smart design same as Hand pump is smart design than basket pull method so we can do easily and get maximum output same is due to smart design of enlightenment on Bodhi Linux. Graphics design, Application minimize, Software installation from browser in Linux, Reliability of Ubuntu Linux, Style of Debian Linux and Power of simple Linux are the best key features of Bodhi Linux.
There are lots of good reviews about Bodhi linux those you can see just type the keywords “bodhi linux” and you will get the lots of good stuff about bodhi linux. 😀 (apne wallon tan poora jor laya dekho ki banda 😀 )

en.wikipedia.org/wiki/Bodhi_Tree

%d bloggers like this: