Category Archives: Ubuntu Linux
Basically there are three parts of Every YAML file.
- MetaData(we define it)
- spec (specifications) (we define that too)
- state(that depennds on kubernetes brain!! which is “etcd!”)
rest is version and kind, version define which API version we are going to use and kind defines the type of deployment or POD we are going to implement here/!!
When you use the Kubernetes API to create the object (either directly or via
kubectl), that API request must include that information as JSON in the request body. Most often, you provide the information to
kubectl in a .yaml file.
kubectl converts the information to JSON when making the API request.
Metadata: It helps uniquely to understand an object, in
UID, and optional
spec – What state you desire for the object
to be continued –> https://cloudplex.io/tutorial/how-to-write-yaml-files-for-kubernetes/
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
Take above yaml file as an example.
Now run Following command to create Deployment.
kubectl create -f nginx-depoy.yaml
Ways to check your pod has been Created or not.
khangura@metal-machine:~$ kubectl get pod NAME READY STATUS RESTARTS AGE hello-minikube-6ddfcc9757-k7qqv 1/1 Running 0 24h nginx-deployment-5d59d67564-57sk7 1/1 Running 0 2m3s nginx-deployment-5d59d67564-6d9t8 1/1 Running 0 2m3s nginx-deployment-5d59d67564-bpsrj 1/1 Running 0 2m3s
We can always Update the deployment like increase/scale number of replicas.
Use following to update the Deployment.
kubectl edit deployment nginx-deployment # it will use our basic editor
Create a Cluster Using MiniKube:
start your Cluster:
$ minikube start # start your Kubernetes cluster
$ kubectl get po -A # get list of all pods and your cluster.
Get to know your cluster.
(base) khangura@metal-machine:~$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-74ff55c5b-zxngm 1/1 Running 1 66d
kube-system etcd-minikube 1/1 Running 1 66d
kube-system kube-apiserver-minikube 1/1 Running 4 66d
kube-system kube-controller-manager-minikube 1/1 Running 1 66d
kube-system kube-proxy-dgvds 1/1 Running 1 66d
kube-system kube-scheduler-minikube 1/1 Running 1 66d
kube-system storage-provisioner 1/1 Running 14 66d
kubernetes-dashboard dashboard-metrics-scraper-c95fcf479-th7gh 1/1 Running 1 66d
kubernetes-dashboard kubernetes-dashboard-6cff4c7c4f-pvs5r 1/1 Running 10 66d
Remember each different service name has different significance.
etcd-minicube – etcd is configuration management system.
apiserver-minicube – for your cluster to interact with clients.
controller-manager-minikube : – A cluster manager node.
there is much ore deep to each node, Need to follow docs to understand things into more deep dive, Right-Now need to remember that Each unit of POD plays significant role in the structure of Cluster.
$ minikube dashboard # To start minikube Dashboard and to analyse how things are going.
$ kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.4
Create a Sample deploymen, Remember that, You can’ expose Pods You need to expose deployent to external posrts.
$ kubectl expose deployment hello-minikube --type=NodePort --port=8080
Link with external port.
You can check status of your deployment.
$ kubectl get services hello-minikube
Use kubectl to forwrd your port.
$ kubectl port-forward service/hello-minikube 7080:8080
Here We are Installing Vanilla Flavour of Kubernetes Cluster(With High Availablility), Which is Small but Production-Ready.
- You need to have at least Two VMs(Master Node and Slave Node)
- Both VMs should connected on the Same Network.
- Master Node must have 2 CPUs and at least 2 GB of RAM.
- Swap must be turned OFF on Both of the VM(s).
- A basic understanding of Linux, Networking and docker unless you are a magician. 😉
Run Following Command on Master Node.(Without Hash 😛 )
# apt install docker.io
# systemctl enable docker
# apt install curl
# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
# apt-add-repository “deb http://apt.kubernetes.io/ kubernetes-xenial main”
# apt install kubeadm
After Running all the Commands above you would be able to Install All Kubernetes Packages on the Master Node.
Now you need to Deploy Cluster. Run Following Commands.
# swapoff -a
# hostnamectl set-hostname master-node
# kubeadm init –pod-network-cidr=10.244.0.0/16 (Look at it’s output Carefully!!)
# mkdir -p $HOME/.kube
# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# chown $(id -u):$(id -g) $HOME/.kube/config
Deploy Pod Network: (There are different Types of Networks you can use. Here we are using flannel)
Add the slave node to the network in order to form a cluster.
Run the Command which you will get in output after running this:
“kubeadm init –pod-network-cidr=10.244.0.0/16” on you Master Node. make sure you have to run Command which will lokk like Following on Your Slave-node. So you can use Kubernetes Cluster.
** Don’t Run Following Command as it is.
kubeadm join 172.31.9.252:6443 –token bixhtx.4bqyhe45g8fvhqeq \
Other Comman Kubernetes commands to Play with your Cluster:
# kubectl get nodes
# kubectl get pods –all-namespaces
Enjoy your Cluster!!
Let’s hamper TimeScale DB over Tea and find out how much Day it can save:
OS: Ubuntu 16.04
CPU/RAM – Third generation i3 / 10 GB
Hyper Threading: Enabled
If this is your First Try with Postgress, make sure You know How to reset Password for Postgress.
Following code we are able to do all the CRUD operations using SQL Language only.
<span id="mce_SELREST_end" style="overflow:hidden;line-height:0;">﻿</span> --creating a simpe table CREATE TABLE conditions ( time TIMESTAMP NOT NULL, location TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL); -- creating a hypertable. now it wil be partioned with location SELECT create_hypertable('conditions','time','location',4); INSERT INTO conditions(time, location, temperature, humidity) VALUES (NOW(), 'office', 70.0, 50.0); SELECT * FROM conditions ORDER BY time DESC LIMIT 100; -- upda<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>ting data on Hyper tables UPDATE conditions SET temperature = 20000, humidity = 50000 WHERE time = '2019-02-17 12:45:02.189422' AND location = 'office'; SELECT * FROM conditions ORDER BY temperature DESC LIMIT 100; -- upserting data -- create it with unique IDs CREATE TABLE conditions_unique ( time TIMESTAMPTZ NOT NULL, location TEXT NOT NULL, temperature DOUBLE PRECISION NULL, humidity DOUBLE PRECISION NULL, UNIQUE (time, location) ); INSERT INTO conditions_unique(time, location, temperature, humidity) VALUES (NOW(), 'office', 70.0, 50.0); INSERT INTO conditions_unique VALUES ('2019-02-17 12:56:05.080302+05:30', 'office', 898900000000.1, 9879678000000000.0) ON CONFLICT DO NOTHING; SELECT * FROM conditions_unique ORDER BY temperature DESC LIMIT 100;
It is as easy to play with bit complex quiries as one is doing Python. 😉
-- Before going with this you have to download and setup NYC data -- Folow this link https://docs.timescale.com/v1.2/tutorials/tutorial-hello-nyc \c nyc_data SELECT date_trunc('day', pickup_datetime) as day, avg(fare_amount) FROM rides WHERE passenger_count > 1 AND pickup_datetime < '2016-01-08' GROUP BY day ORDER BY day; SELECT date_trunc('day', pickup_datetime) as day, COUNT(*) FROM rides GROUP BY day ORDER BY day LIMIT 5; SELECT time_bucket('5 minute', pickup_datetime) AS five_min, count(*) FROM rides WHERE pickup_datetime < '2016-01-01 02:00' GROUP BY five_min ORDER BY five_min; SELECT rate_code, COUNT(vendor_id) as num_trips FROM rides WHERE pickup_datetime < '2016-01-08' GROUP BY rate_code ORDER BY rate_code; -- Join rides with rates to get more information on rate_code SELECT rates.description, COUNT(vendor_id) as num_trips FROM rides JOIN rates on rides.rate_code = rates.rate_code WHERE pickup_datetime < '2016-01-08' GROUP BY rates.description ORDER BY rates.description;
now let's Play with more complex stuff but see how far we can go?
- How large should I configure my intervals for time partitioning?
- Should I use space partitioning, and how many space partitions should I use?
Look for Best Practices:> https://docs.timescale.com/v1.2/using-timescaledb/hypertables
-- ============================================================ -- Structure -- ============================================================ CREATE DATABASE test; \c test; CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE; CREATE TABLE collfact ( time TIMESTAMP NOT NULL, pointid BIGINT NOT NULL, value DOUBLE PRECISION NULL, mesureid BIGINT NULL, version INT NULL, insertiondate TIMESTAMP NULL, iscurrent CHAR(1) ); SELECT create_hypertable('collfact', 'time', 'pointid', 16, chunk_time_interval => interval '1 day'); create index on collfact(time, pointid); -- ============================================================ -- Generate test data, ~ 1 year -- ============================================================ -- meter location, or point create table point( pointid BIGINT NOT NULL PRIMARY KEY, pointnm VARCHAR(20) NOT NULL ); CREATE SEQUENCE pointid_seq START 1; CREATE OR REPLACE FUNCTION genPoints(in a_no_of_records integer) RETURNS integer AS $$ DECLARE v_counter integer; vhash bigint; BEGIN v_counter := 1; RAISE NOTICE 'No of records to insert : %', a_no_of_records; WHILE (v_counter <= a_no_of_records) LOOP IF( v_counter % 10000 =0) THEN RAISE NOTICE 'Counter here is %', v_counter; END IF; vhash := nextval('pointid_seq'); --insert into partiton table INSERT INTO point(pointid, pointnm) VALUES (vhash, 'Point num.' || vhash); v_counter := v_counter + 1; END LOOP; RETURN 0; END; $$ LANGUAGE plpgsql; -- Meter creation select * from genPoints(10000); create table time ( time TIMESTAMP NOT NULL PRIMARY KEY ); -- Year 2018, step 15' INSERT INTO time(time) SELECT dd FROM generate_series (timestamp '2018-01-01 00:00:00' , timestamp '2018-12-31 23:45:00' , '15 minutes'::interval) dd ; -- mesure type table; just 4 here CREATE TABLE mesure ( mesureid BIGINT NOT NULL PRIMARY KEY, mesurenm VARCHAR(20) ); INSERT INTO MESURE (mesureid, mesurenm) VALUES (1, 'P+'); INSERT INTO MESURE (mesureid, mesurenm) VALUES (2, 'P-'); INSERT INTO MESURE (mesureid, mesurenm) VALUES (3, 'Q+'); INSERT INTO MESURE (mesureid, mesurenm) VALUES (4, 'Q-'); \timing on -- INSERT for 1000 meters, 2 mesure types insert into collfact (time, pointid, value, mesureid, version, insertiondate, iscurrent) select time, pointid, trunc(random() * 99999), 1 as mesureid, 1, now(), '1' from (select * from time) as a, (select * from point limit 1000) as b UNION ALL select time, pointid, trunc(random() * 99999), 2 as mesureid, 1, now(), '1' from (select * from time) as a, (select * from point limit 1000) as b; -- INSERT for 1000 meters, 2 mesure types insert into collfact (time, pointid, value, mesureid, version, insertiondate, iscurrent) select time, pointid, trunc(random() * 99999), 3 as mesureid, 1, now(), '1' from (select * from time) as a, (select * from point limit 1000) as b;
I am running out of Shared memory after 15-20 mintues by using above code.
Yo can use code from this Git repo as well:
1. You need to spend time over optmization of DB Setup like how much memory you want to share and should look for how to Optimize PGsql on your System.
2. You must carefully understand your data and how much Chunck-Size you want to define.
3. You don’t have to think much while writing complex Quiries because writing complex quiries is straight forward as well.
4. There is very good CRUD support.
5. You can perform Data-Analysis, Visualization and Machine-Learning very easily because community is supporting it on the FLY.
6. As it is extension of PG-SQL so stability can be defined.
When you don’t want to type username an password each time or when you are not allowed to do so as well.
Your repo should look like this:
you can see config using:
SSH key can be generated like:
ssh-keygen -t rsa -b 4096 -C “email@example.com”
More about Git and adding ssh key can be found here:
This post is intended to be rough draft for preparation for Julia presentation at TCC GNDEC. I am excited.
Now one thing always comes to mind why another Language/Technology?(I am nerd and geek as well, That’s what I do for living passionately!)
There is quite great trend has reached in field of computer science and that is all want to become Data-Scientist. or at least want to get paid as high as possible. DS seems to be right choice. 😉 (this is really a troll 😀 )
First thing how Julia is born?
Someone posted on Reddit!
We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled.
(Did we mention it should be as fast as C?)
WoW!!! that’s real, interactive,compiled,with the speed of c and also easy to learn. as well as general purpose like Python? 😀 I am laughing yeah really laughing.
Well that’s how Julia came to life, or that’s the real motive behind Julia Project.
How you know Julia is for You?
- You are continually involved in computationally intensive work where runtime speed is a major bottleneck. (I just do that with each code block of mine if possible)
2. You use relatively sophisticated algorithms. (sometimes I do that)
3. You write a lot of your own routines from scratch (I don’t do that)
4. Nothing pleases you more than the thought of diving into someone else’s code library and dissecting its internals. (do that ‘a’-lot!)
Most of the above tasks listed I try to Do with ‘Python'(I am actually in serious relationship with this tool 😀 😉 because it never puts me down in Day job and just works! )
HOW JULIA A SO MUCH FASTER????
Julia is a really well-thought-out language. While the syntax looks superficially Matlabby,(Is that really a word?) that is about as far as the similarity goes. Like Matlab, R, and Python, Julia is interactive and dynamically typed, making it easy to get started programming.
But Julia differs from those languages in a few major ways.
Under the hood, it has a rigorous but infinitely flexible type system, and calls functions based on “multiple dispatch”: different code is automatically chosen based on the types of the all arguments supplied to a function.
(is it some kind of skipping type-checks each time?)
When these features are combined with the built-in just-in-time (JIT) compiler,
No GIL.. yeahhh!!!!!
they let code–even scalar for-loops, which are famous performance killers in R–run as fast as C or Fortran.
Yes I want it fast LIke SEE (C)
But the real killer is that you can do this with code as concise and expressive as Python.
I am excited are you to sale the SHIP in the sea?
One thing I am sure about now is there are toys around me whose I need to learn to play.. ❤
REST Python API
WSGI-> well what I know is it is medium between Webserver and your Application.
Application <WSGI> Python-application
It is Linux container, You can use containers to development.
Container seems better options to do things rather than normal..
Dear GOD, Please provide me chance to play with it!
OK first of all if you have no idea what is BIM,IFC,BIMserver than you should be go for some Google and come again. I am writing a guide to setup BIMserver on Ubuntu.
$ sudo apt-get update
$ sudo apt-get install openjdk-7-jdk
$ sudo apt-get install openjdk-7-jre
Set the “JAVA_HOME” environment variable (If you do not know you can Google it or ask in the comments)
Again come to terminal:
$ mkdir BIMserver
$ cp bimserver-1.3.0-FINAL-2014-04-25.jar BIMserver
$ cd BIMserver
$ java -jar bimserver-1.3.0-FINAL-2014-04-25.jar
This will extract the content of the jar file and create two subfolders, i.e. home and bimserver-[version]. Now an application window will open. You have to click on “start server”
Bingo..!! I have not tested it. Let me know(in comments) if you got any problem. Good luck with BIMing. 😉