my net house

WAHEGURU….!

Category Archives: Ubuntu Linux

Understanding YAML in-Depth!!

Basically there are three parts of Every YAML file.

  1. MetaData(we define it)
  2. spec (specifications) (we define that too)
  3. state(that depennds on kubernetes brain!! which is “etcd!”)

rest is version and kind, version define which API version we are going to use and kind defines the type of deployment or POD we are going to implement here/!!

When you use the Kubernetes API to create the object (either directly or via kubectl), that API request must include that information as JSON in the request body. Most often, you provide the information to kubectl in a .yaml file. kubectl converts the information to JSON when making the API request.

Source: https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/

Metadata: It helps uniquely to understand an object,  in

cluding a name string, UID, and optional namespace

spec – What state you desire for the object

to be continued –> https://cloudplex.io/tutorial/how-to-write-yaml-files-for-kubernetes/

RUn Nginx in Kubernetes (Deployment and Scaling replicas) Part-2

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Take above yaml file as an example.

Now run Following command to create Deployment.

kubectl create -f nginx-depoy.yaml

Ways to check your pod has been Created or not.

khangura@metal-machine:~$ kubectl get pod

NAME READY STATUS RESTARTS AGE
hello-minikube-6ddfcc9757-k7qqv 1/1 Running 0 24h
nginx-deployment-5d59d67564-57sk7 1/1 Running 0 2m3s
nginx-deployment-5d59d67564-6d9t8 1/1 Running 0 2m3s
nginx-deployment-5d59d67564-bpsrj 1/1 Running 0 2m3s

We can always Update the deployment like increase/scale number of replicas.

Use following to update the Deployment.

kubectl edit deployment nginx-deployment # it will use our basic editor

Kubernetes Basics(pods creation and understanding)

Create a Cluster Using MiniKube:

https://minikube.sigs.k8s.io/docs/start/

start your Cluster:

$ minikube start # start your Kubernetes cluster

$ kubectl get po -A # get list of all pods and your cluster.

Get to know your cluster.

(base) khangura@metal-machine:~$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-74ff55c5b-zxngm 1/1 Running 1 66d
kube-system etcd-minikube 1/1 Running 1 66d
kube-system kube-apiserver-minikube 1/1 Running 4 66d
kube-system kube-controller-manager-minikube 1/1 Running 1 66d
kube-system kube-proxy-dgvds 1/1 Running 1 66d
kube-system kube-scheduler-minikube 1/1 Running 1 66d
kube-system storage-provisioner 1/1 Running 14 66d
kubernetes-dashboard dashboard-metrics-scraper-c95fcf479-th7gh 1/1 Running 1 66d
kubernetes-dashboard kubernetes-dashboard-6cff4c7c4f-pvs5r 1/1 Running 10 66d

Remember each different service name has different significance.

etcd-minicube – etcd is configuration management system.

apiserver-minicube – for your cluster to interact with clients.

controller-manager-minikube : – A cluster manager node.

there is much ore deep to each node, Need to follow docs to understand things into more deep dive, Right-Now need to remember that Each unit of POD plays significant role in the structure of Cluster.

$ minikube dashboard # To start minikube Dashboard and to analyse how things are going.

$ kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.4

Create a Sample deploymen, Remember that, You can’ expose Pods You need to expose deployent to external posrts.

$ kubectl expose deployment hello-minikube --type=NodePort --port=8080

Link with external port.

You can check status of your deployment.

 $ kubectl get services hello-minikube

Use kubectl to forwrd your port.

$ kubectl port-forward service/hello-minikube 7080:8080

Install Vanilla Kubernetes Cluster on Ubuntu.

Here We are Installing Vanilla Flavour of Kubernetes Cluster(With High Availablility), Which is Small but Production-Ready.

 

Prerequisites:

 

  • You need to have at least Two VMs(Master Node and Slave Node)
  • Both VMs should connected on the Same Network.
  • Master Node must have 2 CPUs and at least 2 GB of RAM.
  • Swap must be turned OFF on Both of the VM(s).
  • A basic understanding of Linux, Networking and docker unless you are a magician. 😉

 

Installation Steps:

Run Following Command on Master Node.(Without Hash 😛 ) 
#  apt install docker.io

# systemctl enable docker

# apt install curl

# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add

# apt-add-repository “deb http://apt.kubernetes.io/ kubernetes-xenial main”

# apt install kubeadm

After Running all the Commands above you would be able to Install All Kubernetes Packages on the Master Node.

Now you need to Deploy Cluster. Run Following Commands. 

 

# swapoff -a

#  hostnamectl set-hostname master-node

# kubeadm init –pod-network-cidr=10.244.0.0/16 (Look at it’s output Carefully!!)

# mkdir -p $HOME/.kube

# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

# chown $(id -u):$(id -g) $HOME/.kube/config

 

 

Deploy Pod Network: (There are different Types of Networks you can use. Here we are using flannel)

 

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

 

Add the slave node to the network in order to form a cluster. 

Run the Command which you will get in output after running this:
kubeadm init –pod-network-cidr=10.244.0.0/16” on you Master Node. make sure you have to run Command which will lokk like Following on Your Slave-node. So you can use Kubernetes Cluster. 

** Don’t Run Following Command as it is.

 

kubeadm join 172.31.9.252:6443 –token bixhtx.4bqyhe45g8fvhqeq \

    –discovery-token-ca-cert-hash sha256:eb5f21bfda175f8d60ca50780941df83169893182c8fd2073de32daccc7b2e6d

 

Other Comman Kubernetes commands to Play with your Cluster:

# kubectl get nodes

# kubectl get pods –all-namespaces

 

Enjoy your Cluster!! 

Hampering TimeScale-DB over Cup of Tea

Let’s hamper TimeScale DB over Tea and find out how much Day it can save:

Hardware Requirements:

OS: Ubuntu 16.04

CPU/RAM – Third generation i3 / 10 GB

Hyper Threading: Enabled

What is TimeScale?

If this is your First Try with Postgress, make sure You know How to reset Password for Postgress.

Following code we are able to do all the CRUD operations using SQL Language only.

 

<span id="mce_SELREST_end" style="overflow:hidden;line-height:0;">&#65279;</span>

--creating a simpe table

CREATE TABLE conditions (
time TIMESTAMP NOT NULL,
location TEXT NOT NULL,
temperature DOUBLE PRECISION NULL,
humidity DOUBLE PRECISION NULL);

-- creating a hypertable. now it wil be partioned with location

SELECT create_hypertable('conditions','time','location',4);

INSERT INTO conditions(time, location, temperature, humidity)
VALUES (NOW(), 'office', 70.0, 50.0);

SELECT * FROM conditions ORDER BY time DESC LIMIT 100;

-- upda<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>ting data on Hyper tables

UPDATE conditions SET temperature = 20000, humidity = 50000
WHERE time = '2019-02-17 12:45:02.189422' AND location = 'office';

SELECT * FROM conditions ORDER BY temperature DESC LIMIT 100;

-- upserting data

-- create it with unique IDs

CREATE TABLE conditions_unique (
time TIMESTAMPTZ NOT NULL,
location TEXT NOT NULL,
temperature DOUBLE PRECISION NULL,
humidity DOUBLE PRECISION NULL,
UNIQUE (time, location)
);
INSERT INTO conditions_unique(time, location, temperature, humidity)
VALUES (NOW(), 'office', 70.0, 50.0);
INSERT INTO conditions_unique
VALUES ('2019-02-17 12:56:05.080302+05:30', 'office', 898900000000.1, 9879678000000000.0)
ON CONFLICT DO NOTHING;

SELECT * FROM conditions_unique ORDER BY temperature DESC LIMIT 100;

It is as easy to play with bit complex quiries as one is doing Python. 😉


-- Before going with this you have to download and setup NYC data
-- Folow this link https://docs.timescale.com/v1.2/tutorials/tutorial-hello-nyc

\c nyc_data

SELECT date_trunc('day', pickup_datetime) as day, avg(fare_amount)
FROM rides
WHERE passenger_count > 1 AND pickup_datetime < '2016-01-08'
GROUP BY day ORDER BY day;

SELECT date_trunc('day', pickup_datetime) as day, COUNT(*) FROM rides
GROUP BY day ORDER BY day
LIMIT 5;

SELECT time_bucket('5 minute', pickup_datetime) AS five_min, count(*)
FROM rides
WHERE pickup_datetime < '2016-01-01 02:00'
GROUP BY five_min ORDER BY five_min;

SELECT rate_code, COUNT(vendor_id) as num_trips FROM rides
WHERE pickup_datetime < '2016-01-08'
GROUP BY rate_code ORDER BY rate_code;

-- Join rides with rates to get more information on rate_code
SELECT rates.description, COUNT(vendor_id) as num_trips FROM rides
JOIN rates on rides.rate_code = rates.rate_code
WHERE pickup_datetime < '2016-01-08'
GROUP BY rates.description ORDER BY rates.description;

now let's Play with more complex stuff but see how far we can go?

  1. How large should I configure my intervals for time partitioning?
  2. Should I use space partitioning, and how many space partitions should I use?

Look for Best Practices:> https://docs.timescale.com/v1.2/using-timescaledb/hypertables


-- ============================================================
-- Structure
-- ============================================================

CREATE DATABASE test;

\c test;

CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;

CREATE TABLE collfact (
time TIMESTAMP NOT NULL,
pointid BIGINT NOT NULL,
value DOUBLE PRECISION NULL,
mesureid BIGINT NULL,
version INT NULL,
insertiondate TIMESTAMP NULL,
iscurrent CHAR(1)
);

SELECT create_hypertable('collfact', 'time', 'pointid', 16, chunk_time_interval => interval '1 day');
create index on collfact(time, pointid);

-- ============================================================
-- Generate test data, ~ 1 year
-- ============================================================

-- meter location, or point
create table point(
pointid BIGINT NOT NULL PRIMARY KEY,
pointnm VARCHAR(20) NOT NULL
);

CREATE SEQUENCE pointid_seq START 1;

CREATE OR REPLACE FUNCTION genPoints(in a_no_of_records integer) RETURNS integer AS $$
DECLARE
v_counter integer;
vhash bigint;
BEGIN
v_counter := 1;
RAISE NOTICE 'No of records to insert : %', a_no_of_records;
WHILE (v_counter <= a_no_of_records) LOOP
IF( v_counter % 10000 =0) THEN
RAISE NOTICE 'Counter here is %', v_counter;
END IF;
vhash := nextval('pointid_seq');

--insert into partiton table
INSERT INTO point(pointid, pointnm)
VALUES (vhash, 'Point num.' || vhash);
v_counter := v_counter + 1;
END LOOP;
RETURN 0;
END;

$$ LANGUAGE plpgsql;

-- Meter creation
select * from genPoints(10000);

create table time (
time TIMESTAMP NOT NULL PRIMARY KEY
);

-- Year 2018, step 15'
INSERT INTO time(time)
SELECT dd
FROM generate_series
(timestamp '2018-01-01 00:00:00'
, timestamp '2018-12-31 23:45:00'
, '15 minutes'::interval) dd ;

-- mesure type table; just 4 here
CREATE TABLE mesure (
mesureid BIGINT NOT NULL PRIMARY KEY,
mesurenm VARCHAR(20)
);

INSERT INTO MESURE (mesureid, mesurenm) VALUES (1, 'P+');
INSERT INTO MESURE (mesureid, mesurenm) VALUES (2, 'P-');
INSERT INTO MESURE (mesureid, mesurenm) VALUES (3, 'Q+');
INSERT INTO MESURE (mesureid, mesurenm) VALUES (4, 'Q-');

\timing on

-- INSERT for 1000 meters, 2 mesure types
insert into collfact (time, pointid, value, mesureid, version, insertiondate, iscurrent)
select time, pointid, trunc(random() * 99999), 1 as mesureid, 1, now(), '1'
from
(select * from time) as a,
(select * from point limit 1000) as b
UNION ALL
select time, pointid, trunc(random() * 99999), 2 as mesureid, 1, now(), '1'
from
(select * from time) as a,
(select * from point limit 1000) as b;

-- INSERT for 1000 meters, 2 mesure types
insert into collfact (time, pointid, value, mesureid, version, insertiondate, iscurrent)
select time, pointid, trunc(random() * 99999), 3 as mesureid, 1, now(), '1'
from
(select * from time) as a,
(select * from point limit 1000) as b;

I am running out of Shared memory after 15-20 mintues by using above code.

Yo can use code from this Git repo as well:

https://github.com/arshpreetsingh/timeScale-analysis

My findings:

1. You need to spend time over optmization of DB Setup like how much memory you want to share and should look for how to Optimize PGsql on your System.

2. You must carefully understand your data and how much Chunck-Size you want to define.

3. You don’t have to think much while writing complex Quiries because writing complex quiries is straight forward as well.

4. There is very good CRUD support.

5. You can perform Data-Analysis, Visualization and Machine-Learning very easily because community is supporting it on the FLY.

6. As it is extension of PG-SQL so stability can be defined.

Some Gitty GITs

When you don’t want to type username an password each time or when you are not allowed to do so as well.

Your repo should look like this:

git@github.com:USERNAME/REPOSITORY.git

you can see config using:


nano .git/config

SSH key can be generated like:

ssh-keygen -t rsa -b 4096 -C “arshpreet.singh@myemail.com”

 

More about Git and adding ssh key can be found here:
https://help.github.com/articles/connecting-to-github-with-ssh/

Julia Why and how not te be Python!(GD Task)

 

 

This post is intended to be rough draft for preparation for Julia presentation at TCC GNDEC. I am excited.

Now one thing always comes to mind why another Language/Technology?(I am nerd and geek as well, That’s what I do for living passionately!)

There is quite great trend has reached in field of computer science and that is all want to become Data-Scientist. or at least want to get paid as high as possible. DS seems to be right choice. 😉 (this is really a troll 😀 )

 

First thing how Julia is born?

Someone posted on Reddit!

 

We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled.

(Did we mention it should be as fast as C?)

 

WoW!!! that’s real, interactive,compiled,with the speed of c and also easy to learn. as well as general purpose like Python? 😀 I am laughing yeah really laughing.

 

Well that’s how Julia came to life, or that’s the real motive behind Julia Project.

 

How you know Julia is for You?

  1. You are continually involved in computationally intensive work where runtime speed is a major bottleneck. (I just do that with each code block of mine if possible)

2. You use relatively sophisticated algorithms. (sometimes I do that)

3. You write a lot of your own routines from scratch (I don’t do that)

4. Nothing pleases you more than the thought of diving into someone else’s code library and dissecting its internals. (do that ‘a’-lot!)

Most of the above tasks listed I try to Do with ‘Python'(I am actually in serious relationship with this tool 😀 😉 because it never puts me down in Day job and just works! )

HOW JULIA A SO MUCH FASTER????

images

 

Julia is a really well-thought-out language. While the syntax looks superficially Matlabby,(Is that really a word?) that is about as far as the similarity goes. Like Matlab, R, and Python, Julia is interactive and dynamically typed, making it easy to get started programming.

But Julia differs from those languages in a few major ways.

Under the hood, it has a rigorous but infinitely flexible type system, and calls functions based on “multiple dispatch”: different code is automatically chosen based on the types of the all arguments supplied to a function.

(is it some kind of skipping type-checks each time?)

 

When these features are combined with the built-in just-in-time (JIT) compiler,

No GIL.. yeahhh!!!!!

they let code–even scalar for-loops, which are famous performance killers in R–run as fast as C or Fortran.

Yes I want it fast LIke SEE (C)

But the real killer is that you can do this with code as concise and expressive as Python.

Still Pythonic!

 

I am excited are you to sale the SHIP in the sea?

 

 

 

Docker and other important things

One thing I am sure about now is there are toys around me whose I need to learn to play.. ❤

REST Python API

Requests module

Docker

Django

WSGI-> well what I know is it is medium between Webserver and your Application.

Application <WSGI> Python-application

Docker… Why?

It is Linux container, You can use containers to development.

Container seems better options to do things rather than normal..

Dear GOD, Please provide me chance to play with it!

Install BIMserver on Ubuntu

OK first of all if you have no idea what is BIM,IFC,BIMserver than you should be go for some Google and come again. I am writing a guide to setup BIMserver on Ubuntu.

$ sudo apt-get update
$ sudo apt-get install openjdk-7-jdk
$ sudo apt-get install openjdk-7-jre

Set the “JAVA_HOME” environment variable (If you do not know you can Google it or ask in the comments)

Again come to terminal:

$ wget https://github.com/opensourceBIM/BIMserver/releases/download/1.3.0-FINAL-2014-04-25/bimserver-1.3.0-FINAL-2014-04-25.jar

$ mkdir BIMserver

$ cp bimserver-1.3.0-FINAL-2014-04-25.jar BIMserver

$ cd BIMserver

$ java -jar bimserver-1.3.0-FINAL-2014-04-25.jar

This will extract the content of the jar file and create two subfolders, i.e. home and bimserver-[version]. Now an application window will open. You have to click on “start server”
Bingo..!! I have not tested it. Let me know(in comments) if you got any problem. Good luck with BIMing. 😉

%d bloggers like this: