my net house

WAHEGURU….!

Asynchronous recipes in Python

“Concurrency” is not “Parallelism” May be it’s better. If you will not work with DataScience, DataProcessing, Machine-Learning and other operations which are CPU-Intensive you probably will found that you don’t need parallelism but you need concurrency more!

  1. A Simple Example is Training a machine learning model is CPU intensive or You can use GPU.
  2. To Make various Predictions from one model based on many Different Input-Parameters to find out best result You need Concurrency!

There are so Many ways one can hack into Python stuff and do cool Stuff either it is CPU intensive or just a task to do stuff that is good/bad/Better/Best for one user to communicate. One thing you have to believe that Python Does support Multiprocessing as well as Multi-threading

but for various reasons when you are doing CPU intensive Tasks you have to Stay away from using Threading operations in Python. Use Numpy, Cython,Jython or anything you feel, Write C++ code and glue it with Python

The number of threads will usually be equivalent to the number of cores you have. If you have hyperthreading on your processor, than you will be able to double the number of threads used.

Above image is just one Example to understand what actually we are doing. We are processing Chunks and Chuncks of Data. Now the real common scenario is If you are using I/O bound tasks use Threads in Python if you are using CPU bound tasks use Processes in Python.  I have worked with various Python Projects where Performance was issue at some level so at that time I always went to other things like Numpy, Pandas, Cyhton or numba but not Plain-Python.

Let’s come to the point and Point is What are those Recipes I can use:

Using concurrent.futures(futures module is also back-ported into Python2.x):

Suppose you have to call multiple URLs at same time using same Method. That is what actually Concurrency is, Apply same method different operations, We can do it either using ThreadPool or ProcessPool.

# Using Process Pool
from concurrent.futures import ProcessPoolExecutor,as_completed
def health_check1(urls_list):
pool = ProcessPoolExecutor(len(urls_list))
futures = [pool.submit(requests.get,url,verify=False) for url in final_url]
results = [r.result() for r in as_completed(futures)] # when all operations done
return results # a Python list of all results, Here you can also use Numpy as well

Using ThreadPool it is also not different:

# Using Thread Pool
from concurrent.futures import ThreadPoolExecutor,as_completed</code>

def just_func(urls_list):
pool = ThreadPoolExecutor(len(urls_list))
futures = [pool.submit(requests.get,url,verify=False) for url in urls_list]
results = [r.result() for r in as_completed(futures)] # when all operations done
return results # a Python list of all results, Here you can also use Numpy as well

In the above code ‘url_list’ is just list of tasks which are similar and can be processed using same kind of functions.

On the other-side using it with with as context manager is also not different. In this Example I will Use ProcessPoolexecutor’s inbuilt map function.

def just_func(url_list):
   with concurrent.futures.ProcessPoolExecutor(max_workers=len(final_url)) as executor:
        result = executor.map(get_response,final_url)
    return [i for i in result]

Using multiprocessing: (Multiprocessing is also Python-library that can be used for Asynchronous behavior of your code.)

*in Multiprocessing the difference between map and apply_async is only that Map returns results as task list is passed to it on the other-hand apply_async returns results based on results those returned by function.

# Function that run multiple tasks
def get_response(url):
“””returns response for URL ”””
    response = requests.get((url),verify=False)
   return response.text

Now above function is simple enough that is getting one URL and returning response but if have to pass multiple URLs but I want that get request to each URL should be fired at same time then That would be Asynchronous process not multiprocessing because in Multiprocessing Threads/Processes needs to communicate with each other but on the other hand in case of Asynchrounous threads don’t communicate(in Python because Python uses Process based multiprocessing not Thread Based although you can do thread-based multiprocessing in Python but then you are on your OWN 😀 😛 Hail GIL (Mogambo/Hitler)).

So above function will be like this as usual:

from multiprocessing import Pool
pool = Pool(processes=20)
resp_pool = pool.map(get_response,tasks)
URL_list = []
resp_pool = _pool.map(get_response,tasks)
pool.terminate()
pool.join()

Although This is an interesting link one can watch while going into Multiprocessing in Python using Multiprocessing: It is Process-Bases Parallelism.
http://sebastianraschka.com/Articles/2014_multiprocessing.html

Using Gevent: Gevent is a concurrency library based around libev. It provides a clean API for a variety of concurrency and network related tasks.

import gevent
import random

def task(pid):
"""
Some non-deterministic task
"""
    gevent.sleep(random.randint(0,2)*0.001)
    print('Task %s done' % pid)

def asynchronous():
    threads = [gevent.spawn(task, i) for i in xrange(10)]
    gevent.joinall(threads)

    print('Asynchronous:')
asynchronous()

If you have to Call Asynchronously but want to return results in Synchronous Fashion:

import gevent.monkey
gevent.monkey.patch_socket()

import gevent
import urllib2
import simplejson as json

def fetch(pid):
    response = urllib2.urlopen('http://json-time.appspot.com/time.json')
    result = response.read()
    json_result = json.loads(result)
    datetime = json_result['datetime']

    print('Process %s: %s' % (pid, datetime))
    return json_result['datetime']

def asynchronous():
    threads = []
    for i in range(1,10):
        threads.append(gevent.spawn(fetch, i))
    gevent.joinall(threads)

print('Asynchronous:')
asynchronous()

Assigning Jobs in Queue:

import gevent
from gevent.queue import Queue</code>

tasks = Queue()

def worker(n):
    while not tasks.empty():
        task = tasks.get()
        print('Worker %s got task %s' % (n, task))
   gevent.sleep(1)

print('Quitting time!')

def boss():
   for i in xrange(1,25):
      tasks.put_nowait(i)

    gevent.spawn(boss).join()

   gevent.joinall([
gevent.spawn(worker, 'steve'),
gevent.spawn(worker, 'john'),
gevent.spawn(worker, 'nancy'),
])

When you have to manage Different Groups of Asynchronous Tasks:

import gevent
from gevent.pool import Group</code>

def talk(msg):
    for i in xrange(3):
        print(msg)

g1 = gevent.spawn(talk, 'bar')
g2 = gevent.spawn(talk, 'foo')
g3 = gevent.spawn(talk, 'fizz')

group = Group()
group.add(g1)
group.add(g2)
group.join()

group.add(g3)
group.join()

Same As multiprocessing Library you can also use Pool to map various operations:

import gevent
from gevent.pool import Pool</code>

pool = Pool(2)

def hello_from(n):
    print('Size of pool %s' % len(pool))

pool.map(hello_from, xrange(3))

Using Asyncio:

Now let’s talk about concurrency Again! There is already lot of automation is going inside asyncio or Gevent but as programmer we have to understand how we need to break a “One large task into small chuncks of Subtasks so when we will write code we will be able to understand which tasks can work independently.

import time
import asyncio

start = time.time()

def tic():
    return 'at %1.1f seconds' % (time.time() - start)

async def gr1():
# Busy waits for a second, but we don't want to stick around...
    print('gr1 started work: {}'.format(tic()))
    await asyncio.sleep(2)
    print('gr1 ended work: {}'.format(tic()))

async def gr2():
# Busy waits for a second, but we don't want to stick around...
    print('gr2 started work: {}'.format(tic()))
    await asyncio.sleep(2)
    print('gr2 Ended work: {}'.format(tic()))

async def gr3():
    print("Let's do some stuff while the coroutines are blocked, {}".format(tic()))
    await asyncio.sleep(1)
    print("Done!")

ioloop = asyncio.get_event_loop()
tasks = [
ioloop.create_task(gr1()),
ioloop.create_task(gr2()),
ioloop.create_task(gr3())
]
ioloop.run_until_complete(asyncio.wait(tasks))
ioloop.close()

Now in the above code gr1 and gr2 are somehow taking some time to return anything it could any kind of i/o operation so what we can do here is go to the gr3 in using the event_loop and event_loop will run until all three tasks are not completed.

Please have a closer look at await keyword in the above code. It is one of the most important step where you can assume interpreter is shifting from one task to another or you can call it pause for function. If you have worked with yield or yield from in Python2 and Python3 you would be able to understand that this is stateless step for the code.

There is on more library which is aiohttp that is being used to handle blocking Http requests with asyncio.

import time
import asyncio
import aiohttp

URL = 'https://api.github.com/events'
MAX_CLIENTS = 3

async def fetch_async(pid):
    print('Fetch async process {} started'.format(pid))
    start = time.time()
    response = await aiohttp.request('GET', URL)
    return response

async def asynchronous():
    start = time.time()
   tasks = [asyncio.ensure_future(fetch_async(i)) for i in range(1, MAX_CLIENTS +1)]
   await asyncio.wait(tasks)
    print("Process took: {:.2f} seconds".format(time.time() - start))

print('Asynchronous:')
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(asynchronous())
ioloop.close()

In all the above Examples we have just Scratched the world of concurrency but in real there would be much more to look into because real world problems are more complex and intensive. There are various other options in asyncio like handling exceptions with-in futures, creating future wrappers for normal tasks,Applying timeouts if task is taking more than required time and doing something else instead.

There is lot of inspiration I got while learning about concurrent programming in Python from the following Sources:

https://hackernoon.com/asyncio-for-the-working-python-developer-5c468e6e2e8e
http://www.gevent.org/
https://www.binpress.com/tutorial/simple-python-parallelism/121
http://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.html

Advertisements

Run Flask in Parallel using ThreadPoolExecutor

from flask import Flask
from time import sleep
from concurrent.futures import ThreadPoolExecutor

# DOCS https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
executor = ThreadPoolExecutor(2)

app = Flask(__name__)

@app.route('/jobs')
def run_jobs():
executor.submit(some_long_task1)
executor.submit(some_long_task2, 'hello', 123)
return 'Two jobs was launched in background!'

def some_long_task1():
print("Task #1 started!")
sleep(10)
print("Task #1 is done!")

def some_long_task2(arg1, arg2):
print("Task #2 started with args: %s %s!" % (arg1, arg2))
sleep(5)
print("Task #2 is done!")

if __name__ == '__main__':
app.run()

OOPS and More OOPS in Python

Concurrency in Python or Natural way of life(Not yet completed POST)

There are various ways one can think about computing , Multiprocessing, Asynchronous, Multi-threading as well as “Parallel Processing” If I would talk about theoratical things I Would say we have to distribute our one particular task in various forms so multiple resources should be available for system to run things or in other way we can say multiprocessing is more of Programmer’s way of understanding the Flow of precess and sometimes rules according to theory does not assure that if one is providing multiple resources to process it will be FAST! it could be FAT! also.

Now let me start with very simple Example by taking following function as use case:

# Function that run multiple tasks
def get_response(url):
“””returns response for URL ”””
response = requests.get((url),verify=False)
return response.text

Now above function is simple enough that is getting one URL and returning response but if I have to pass multiple URLs but I want that get request to each URL should be fired at same time then That would be Asynchronous process not multiprocessing because in Multiprocessing Threads/Processes needs to communicate with each other but on the other hand in case of Asynchrounous threads don’t communicate(in Python because Python uses Process based multiprocessing not Thread Based although you can do thread-based multiprocessing in Python but then you are on your OWN 😀 😛 Hail GIL (Mogambo/Hitler)).

So above function will be like this as usual:

from multiprocessing import Pool
pool = Pool(processes=20)
resp_pool = pool.map(get_response,tasks)
URL_list = []
resp_pool = _pool.map(get_response,tasks)
pool.terminate()
pool.join()

One thing you have to understand very carefully and that is GIL does not harm for i/o bound operations but yes when it comes to non-i/o bound operations in python You have Numpy,Scipy,Pandas,Cython where one can really release GIL and take full advantage of the code.

How to release GIL using Cython: https://lbolla.info/blog/2013/12/23/python-threads-cython-gil
Although one can look for interesting features about GIL: http://www.dabeaz.com/python/NewGIL.pdf

Intel has also provided Python Distribution that is helpful get speedups in Python but that would only be helpful for Machine-learning and Data-Science work.

http://www.techenablement.com/orders-magnitude-performance-intel-distribution-python/(Seems like worth to give it a Try:::)

Now there is one important thing you must need to care about when you are releasing GIL in Python.

You can also scratch your head many times by just reading/watching this one interesting presentation: http://www.dabeaz.com/python/UnderstandingGIL.pdf

Although Numba is also there but make one thing for sure Use such tools only when your Operation is CPU bound not I/O bound because as I have stated that I/O bound operations don’t care about GIL.

Although you will find out that GIL is not just Python’s Problem:

https://www.jstorimer.com/blogs/workingwithcode/8085491-nobody-understands-the-gil

I/O Bound:

The I/O bound state has been identified as a problem in computing almost since its inception. The Von Neumann architecture, which is employed by many computing devices, is based on a logically separate central processor unit which requests data from main memory,[clarification needed] processes it and writes back the results. Since data must be moved between the CPU and memory along a bus which has a limited data transfer rate, there exists a condition that is known as the Von Neumann bottleneck. Put simply, this means that the data bandwidth between the CPU and memory tends to limit the overall speed of computation. In terms of the actual technology that makes up a computer, the Von Neumann Bottleneck predicts that it is easier to make the CPU perform calculations faster than it is to supply it with data at the necessary rate for this to be possible.

In simple cases CPU is Faster and Memory is Slower.
https://en.wikipedia.org/wiki/I/O_bound

Let’s make things more precise:
Sync: Blocking operations.
Async: Non blocking operations.
Concurrency: Making progress together.
Parallelism: Making progress in parallel.

Now Questions arises that do we need all those things together:
http://docs.python-guide.org/en/latest/scenarios/speed/
https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html
https://github.com/dask/dask(Although I just found that Dask is much more Advanced and Promising that one should not ignore at all!!)
http://dask.pydata.org/en/latest/dataframe-performance.html

async: https://hackernoon.com/asyncio-for-the-working-python-developer-5c468e6e2e8e
https://stackoverflow.com/questions/8533318/python-multiprocessing-pool-when-to-use-apply-apply-async-or-map
https://github.com/pyparallel/pyparallel

Running Multiprocessing in Flask App(Let’s Spawn) Hell Yeah

Ok It was going to be long time but Finally yeah Finally Able to do Process based multiprocessing in Python and even on Flask. 🙂 oh yeah! There are various recipes for Multiprocessing in this python but here you can only Enjoy with Flask.

:D


from multiprocessing import Pool
from flask import Flask
from flask import jsonify
import ast
import pandas as pd
import requests

app = Flask(__name__)
_pool = None

# Function that run multiple tasks
def get_response(x):
"""returns response for URL list"""
m = requests.get((x),verify=False)
return m.text

@app.route('/call-me/')
def health_check():
"""returns pandas dataframe into HTML for health-check Services"""
resp_pool = _pool.map(get_response,tasks)
table_frame= pd.DataFrame([ast.literal_eval(resp) for resp in resp_pool])
return table_frame.to_html()

if __name__=='__main__':
_pool = Pool(processes=12) # this is important part- We
try:
# insert production server deployment code
app.run(use_reloader=True)
except KeyboardInterrupt:
_pool.close()
_pool.join()

 

One mintue read to one minute Manager

Get out more results in less time.

Autocratic VS Democratic:
Autocratic are result oriented and Democratic are happiness Oriented So we
need to be one minute Managers. 🙂

1. One minute Goal Setting:

Everyone should be knowing the goals of the company.
People must know what their roles are in the company.
Goals must not be more than 250 words.
Always review your Goals.

2. One minute Praising:

Give True Feedback.
Always praise immediately.
Share happiness and encourage your people.

3. One minute reprimand:

Immediately point people out for their mistakes.
Tell people how you feel about it.
Point out mistake but don’t criticize.
Be on the side of your people.

conclusion:
Look for the good things in the beginners and bad things in the experienced.
Share what you learn.
We don’t manage people, We manage behaviors.
Love your people and make sure they are also loving you back.
Define your problem grammatically. (What is happening and What you want to be happen.)

OOPS Design Principles(Completing course)

Writing a Method that takes one or few parameters as input and then returns various Outputs depends on the logic of the  function. This post will be dedicated on the OOPS design Principles and how those somehow complete Design principles for software development. All the principles in SOLID are somehow connected with each other and one can follows any one of those for Better development of code. Follow any principle(Say Open-Close) and after some time of developing your code you will find that you actually are following all the principles. 🙂 Because all these principles are developed by making one thing in mind and that is High-Quality Software development.

SOLID:

S(Solid Responsibility Principle): There should never be more than one reason to for a class to change.

O(Open-Close Principle): Classes(Modules or functions) should be open for extensions but closed for modifications.

***Features should be parameterized in the way that class can be overridden.

OCP suggests that some bug fixes should be viewed as an extensions instead of a modifications.

That clarifies the single responsibility principle. A class should be able to do only on kind of task. Kind means Reading data. Data could be read from one type of file or many type of files. For example there could be class name DataReader() and it could have many methods. csv_reader(), excel_reader(), db_reader(), txt_reader() so now single-responsibility of class is to ‘read data’ 🙂 any kind of data, by any means but read data. 🙂

Any class that we are going to write should be written as is it could be extended but no need to make new changes in the code of the class.

Interface Segregation Principle: (I from SOLID)

*** No client should be forced to depend on methods it does not use.
*** Class should be designed so collaborators should have narrowest interface.
*** If you are using Python Unit Test cases should be there because we are using Python!

Classes that implement interfaces should not be forced to implement methods they(classes) do not use. (Use small interfaces not FAT ones) Do not create fat interfaces.
We need different streams for read and write.

A client should be dependent on smallest set of Methods and attributes. The fewest methods or attributes.

Segregation defines parting the functions(Not programming functions) of Software into two or more different classes for ease of client to access the features.

D: Dependency inversion principle.

*** Applications should use abstract class code that lead to injection.
*** In language like Python we can use settings.py file to perform tasks like this one.

High level modules should not be dependent on low level modules. Abstractions should not be dependent on Details, Details should depend on abstractions. There should be one ‘Abstraction-Layer’ on each of the module so one would be able to test the behavior of classes without even looking at the classes.

L:(Liskov substitution Principle):

*** Behavior of sub-class should be as correct as behavior of super class.

When you will follow the Liskov substitution principle your classes will follow Open/Close principles implicitly.

Substitutability is a principle in object-oriented programming stating that, in a computer program, if S is a subtype of T, then objects of type T may be replaced with objects of type S (i.e. an object of type T may be substituted with any object of a subtype S) without altering any of the desirable properties of T (correctness, task performed, etc.).

Single responsibility principle:

** A class should have only one reason to change.

GRASP:>>>> General Responsibility assignment principle.
**Controller and Creation
**High Cohesion and Indirection
** High Cohesion:
Features those belong together.
High Cohesion are combined into single responsibility.
** Hard part is locating level of abstractions.

Why we should think about GRASP?:
Information Expert
Pure Fabrication
Low coupling
Polymorphism
Protected variations

Sources: https://www.lynda.com/Programming-Languages-tutorials/GRASP-patterns/471978/502212-4.html

Lessons Learned from life

Complete Basics those just went out of my mind, No idea how those gone away. 😦

Work-Life Balance.

You can’t be successful in one day.

All the time people around you tell how to do it, Either you ignore  it or take to next level.

Better late than never.

Never Leave your Day job(even if it is cutting grass ).

Don’t try to be OVER-SMART.

Never consume any Addictive substance.

Learn to respect your personal space as well as others.

Learn to turn off your mind from consistence thinking of things.

Love your work—-Work is never Ending process, Don’t take so much pressure to complete it or start next one.

Have a group of friends outside work.

Nobody is slowing you down Except you.

Learn to say sorry, please, thanks, welcome.

Help others but respect your time and Energy.

Break the pattern of your life.

Be hungry, be foolish – Stop believing that.

Sikhism has different way of living life.(Either believe in that or Live with sorrows.)

If you want to earn more, Be-crazy, Get-exploited and create a big hole inside you, that is your choice as well. 🙂

Python For Yankees

Before going further reading about Class Development I would like you to tell
one important thing:

Most of the uses of inheritance can be simplifi ed or replaced with composition, and multiple
inheritance should be avoided at all costs.

I am sure after this you will be able to read lots and lots of code written in
python. 🙂 If you want to do it fast, Do it well. 🙂

Python Class Development Toolkit.

def __init__():
    pass

#Don't put anything in instance that you don't need instance for.

def i_am_class_method(self):
    return 'takes self as argument'

#If you need to shared data for whole class then put it on class level,
#not on instance level.

class IAm(object):
    shared_data=[1,2,3,4,5]

    def __init__(self):
        print shared_data

    def fun_2(self):
        print shared_data

Iron_clad_rule:- in java or C++ is–>do not expose your attributes

Subclassing is just like inhertance. Data+methods:

Class NewOne(IAm):

    def fun3(self):
       return IAm.fun_2(self)

#Multiple/Alternative-Constructors:(When you need to change the behavior
#of class's data by just calling an intensive function):

class Circle(object):

    def init_(self,radius):
        self.radius=radius

    def fun_2(self):
        print self.radius

   @classmethod   # Alternative constructive
   def from_bbd(cls,new_radius):
        """construct a circle from bounding box diagnol"""

       raduis = new_radius/2.0/math.sqrt(2.0)
       return Circle(radius)

Make Alternate constructor to work for Subclass as well:

class Circle(object):

    def __init__(self,radius):
       self.radius=radius

    def fun_2(self):
        print self.radius

   @classmethod   # Alternative constructive
    def from_bbd(cls,new_radius):
       """construct a circle from bounding box diagnol"""
        raduis = new_radius/2.0/math.sqrt(2.0)
        return cls(radius)

Independent methods inside class: – Why we ever need those?
Think of situation where everything in your code breaks but you just need to tell
user something and something and all you need is 🙂 Static-Method.!!

class Circle(object):

    def __init__(self,radius):
        self.radius=radius

    def fun_2(self):
        print self.radius

    @staticmthod
    def just_method(just_parameter):
        return 'I am independent'

Getters and Setters in Python:(Access Data,change data on the fly)
As in the other languages there are inbuilt methods to access data from
class as well as change values of method attributes in the class.

Python uses the property decorator to perform such operations:

class Person(object):
    def __init__(self, first_name, last_name):
       self.first_name = first_name
       self.last_name = last_name</code>

    @property
    def full_name(self):
        return self.first_name + ' ' + self.last_name

    @full_name.setter
    def full_name(self, value):
        first_name, last_name = value.split(' ')
        self.first_name = first_name
        self.last_name = last_name

    @full_name.deleter
    def full_name(self):
       del self.first_name
       del self.last_name

Slots in Pyhton:
When you have lots and lots of things to perform or do in the way that
your class instance is consuming HUGE memory then you must use SLOTS.

Earlier approach:

class MyClass(object):

def __init__(self, name, identifier):
    self.name = name
    self.identifier = identifier
    self.set_up()

#Approach after SLOTS:

class MyClass(object):
    __slots__ = ['name', 'identifier']
    def __init__(self, name, identifier):
       self.name = name
       self.identifier = identifier
       self.set_up()

Law of Wealthy LIfe

Wealthy life does not just mean to having lots of money in the bank but it is much more like creating various things in your society or running various engines those work in the manner that you are really able to make things happen in your life instantly, One thing you must remember or know carefully and that is If you really want to do it fast,  do it well. 🙂

Speed of implementation

Respect your time(Don’t waste on social media and stuff)

Go to bed early and get up early. Although I am writing this post so Late.:P 😦 😉

%d bloggers like this: