my net house

WAHEGURU….!

Category Archives: Enlightenment

TPOT Python Example to Build Pipeline for AAPL

This is  just first Quick and Fast Post.

TPOT Research  Paper: https://arxiv.org/pdf/1702.01780.pdf


import datetime
import numpy as np
import pandas as pd
import sklearn
from pandas_datareader import data as read_data
from tpot import TPOTClassifier
from sklearn.model_selection import train_test_split

apple_data = read_data.get_data_yahoo("AAPL")
df = pd.DataFrame(index=apple_data.index)
df['price']=apple_data.Open
df['daily_returns']=df['price'].pct_change().fillna(0.0001)
df['multiple_day_returns'] =  df['price'].pct_change(3)
df['rolling_mean'] = df['daily_returns'].rolling(window = 4,center=False).mean()

df['time_lagged'] = df['price']-df['price'].shift(-2)

df['direction'] = np.sign(df['daily_returns'])
Y = df['direction']
X=df[['price','daily_returns','multiple_day_returns','rolling_mean']].fillna(0.0001)

X_train, X_test, y_train, y_test = train_test_split(X,Y,train_size=0.75, test_size=0.25)

tpot = TPOTClassifier(generations=50, population_size=50, verbosity=2)
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
tpot.export('tpot_aapl_pipeline.py')

The Python file It returned: Which is real Code one can use to Create Trading Strategy. TPOT helped to Selected Algorithms and Value of It’s features. right now we have only provided ‘price’,’daily_returns’,’multiple_day_returns’,’rolling_mean’ to predict Target. One can use multiple features and implement as per the requirement.


import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

# NOTE: Make sure that the class is labeled 'target' in the data file
tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)
features = tpot_data.drop('target', axis=1).values
training_features, testing_features, training_target, testing_target = \
            train_test_split(features, tpot_data['target'].values, random_state=42)

# Score on the training set was:1.0
exported_pipeline = GradientBoostingClassifier(learning_rate=0.5, max_depth=7, max_features=0.7500000000000001, min_samples_leaf=11, min_samples_split=12, n_estimators=100, subsample=0.7500000000000001)

exported_pipeline.fit(training_features, training_target)
results = exported_pipeline.predict(testing_features)

Advertisements

Socket Programming and have fun with Python

Client Socket and Server Socket:

Client Computer like your browser or any piece of code you want to talk to your server uses client socket and Server uses both client and server socket.

Sockets are great for Cross-Platform communication.

Following is minimal Example of Socket and stuff:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("www.python.org", 80))

What is INET?What is Sock_Stream?

Almost that is all happened on client side, When connect is completed socket that is ‘s’ we just created can be used to send and request the specific text page requested. This socket will be read and reply, after that it will be destroyed. Client sockets are normally only used for one exchange (or a small set of sequential exchanges).

Now let’s look what is happening at server side:

# create an INET, STREAMing socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# bind the socket to a public host, and a well-known port
serversocket.bind((socket.gethostname(), 80))
# become a server socket
serversocket.listen(5)

 

Generators, Context-Managers and Coroutines:(Completing course)

Generator Pipelines:

  1. Several Pipelines can be linked together.
  2. Items flows one by one through the entire Pipeline.
  3. Pipeline Functionality can be packaged into callable functions.

One Example of Generator pipeline:

def seprate_names():

for full_name in names:

for name in full_name.split(‘ ‘):

yield name

full_names = name.strip() for name in open(‘names.txt’)

names = seprate_names(full_names)

lengths = ((name,len(name))for name in names)

longest  = max(lengths,key=lambda x:x[1])

Another approach is as follows is one wants to use using Function name: anotherapproac

 

Context-Manager:

Why we need context-Manager?

‘with’ statement in Python that we use to do file operations is Context-manager. It is something like to “Have the state and open that state and with-in that state to do things”. using with in Python we open file and till the file is open we do some good things and after doing all good things we close the file. So ‘with’ is a context Manager using that we make the state of file open and after that we do all the things we need to do with file.

Other useful cases of Context-manager:

Close/Open File/Socket(Even it crashes)

Commit/Fetch (Even if crashes)

release the lock (Even it crashes)

When you really need Context-Managers?

At last fun not least, If I will be creating a Chat-BOT in Python then I would be able to use Context-Manager in Python so I would be doing some stuff and after completing that stuff I can go out and have fun.

when_need_context?

So we use @staticmethod in Python class,  that means no matter what happens we will be able to run this method at any cost.

That is just using decorator. Now if we want to create a context manager using decorator?

So what is Context-manager?

  1. I have to go to a particular directory, list all files with .txt extension, then come-back to current location (Simple use case)

context

  1. I have to load specific ML model, I have to predict against several parameters and get results, Return at specific state(un-load the model)
  2. I have to open Socket connection, Read various kind of data, close Socket connection.

On the other way we can also write it like:

Setup

Yield-control

Wrap-UP

Context-managers are powerful tool to make code more modular and Succinct.

Now What if we have to use Context-manager as Yielded value?

interesting_context

May be little-bit  more about COntext-managers-

https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/

What are Coroutines and how we have to handle those?

  1. Receive Values
  2. May not return anything
  3. Not for iteration

What is the design of Co-routine:

  1. Receive Input
  2. Process that Input
  3. Stop at yield statement

send() method is used to send value to coroutines.

 

More uses of Co-routines:

Coroutines are really powerful for Data-Processing Operations.

https://stackoverflow.com/questions/303760/what-are-use-cases-for-a-coroutine(Concurrency and Multi-Processing)

One of most important course about Co-Routines/Concurrency and really interesting way to handle multiprocessing: http://www.dabeaz.com/coroutines/ —think win win—-

 

Common Regular Expressions

Find line in multiple files:


grep -rnw '/home/ubuntu/workspace/tools-tpn-ops/' -e 'import os'

case insensitive: grep -i "Aditi" helloaditi

case ins and separate existance: grep -iw "is" helloaditi

match and show three lines after that: grep -A 3 -i "am" helloaditi

case of exclusion: grep -v -e "arsh" -e "honey" helloaditi

find position of word in file: grep -o -b "aditi" helloaditi

find line number of matching text: grep -n "sharma" helloaditi

read a specified line number: if you exactly know the line number then do

lets say you want to read line 84
head -84 a.txt| tail -1

Engineer’s Guide to Julia Programming

Engineer’s Guide to Julia Programming

Finally the moment has come when I can say that I can be productive as well as my solution can be Parallel,Optimize-able,Customizable and at last but not least glue-able. Yes those are the fantastic features I believe one can rely on while Learning any New Programming language and Developing a Very High Quality AI/ML Embedded Software Solution.

Why?

Julia Solves Two Language Problem.

Important Disclaimer for Newbies: I am Pythonista by choice and over the last few years I have Developed Projects using Python and it’s sister technologies to provide the solutions those are related to

Automation(Python -Scripting)

Web-Development(Django,Flask,Sanic,Tornado)

Data Analysis(SageMath,Sympy,Paraview,Spread-Sheets,Matplotlib,Numpy,Scipy,SKLearn)

Quantitative-Analysis(Quantopian.com)

3D Modeling(FreeCad, BIM,IFC), and Cluster Computing(Rock’s Cluster).

Now I just wanted a tool that would allow me to write Pure-mathematical expressions(using required signs not variable names) and write Machine-Learning/Artificial-Intelligence/Deep-Learning code where I would find myself on core layer of abstraction not like Tensor-Flow, Pytorch, or Numpy/Pandas. I am not against these libraries those helped me “soooo” much over the years but I have no idea that what kind of things are happening under the hood and may be I will never be allowed to change the working internals of Numpy/Pandas/Cython or anything that is related to Scientific Python only because there could be large amount of Fortran/C++ or Pascal kind things and crunching numbers as well.

Stuff that an Engineer need to perform for various kinds of jobs in Julia-Programming Language can be described as follows:

Solving a Simple Mathematical Equation in Julia:

A = randn(4,4)
x = rand(4)
b = A*x
x̂ = A\b # here we have written x-hat Symbol
println(A)
println(x)
println(x̂)
@show norm(A*x̂ - b)

Doing Matirx Operations in Julia

A = randn(4,4) |> w -> w + w' # pipe A through map A -> A + A' to symmetrize it
println(A)
λ = eigmax(A); # have you checked the lambda?
@show det(A - λ*I)

Performing Integration:

Performing integration might be one of the most important task one would be doing in Day to day if someone is involved with problems related to modeling and designing a solution using CAS(Compute Algebraic System) like Matlab or Sage-Math but designing a solution using CAS and then finding various ways to implement it into production is kind of “LOt of WoRk” I assume that only come with Either Experience or Lots of Extra Brain cells. 😉 See here Julia Plays an important role: “Solving two Language Problem”.

# Integrating Lorenz equations
using ODE
using PyPlot
# define Lorenz equations
function f(t, x)
σ = 10
β = 8/3
ρ = 28
[σ*(x[2]-x[1]); x[1]*(ρ-x[3]); x[1]*x[2] - β*x[3]]
end

# run f once
f(0, [0; 0; 0])

# integrate
t = 0:0.01:20.0
x₀ = [0.1; 0.0; 0.0]
t,x = ode45(f, x₀, t)

x = hcat(x…)’ # rearrange storage of x

# Side-Note::: What … is doing in Julia? (Remember *args and **kwargs in Python?)
# for more see:
goo.gl/mTmeR7

# plot
plot3D(x[:,1], x[:,2], x[:,3], “b-”)
xlabel(“x”)
ylabel(“y”)
zlabel(“z”)
xlim(-25,25)
ylim(-25,25)
zlim(0,60);

Really interesting Dynamic Type System()::

This is one of the most interesting part for me to have so much fun with Julia and it’s GREAT! Type System, You know why? Because It knows how long that bone is and how much Calcium will be there:

### Built-in numeric types

Julia’s built-in numeric types include a wide range of
1. integers: Int16, Int32, Int64 (and unsigned ints), and arbitrary-precision BigInts
2. floating-points: Float16, Float32, Float64, and arbitrary-precision BigFloats
3. rationals using the integer types
4. complex numbers formed from above
5. vectors, matrices, linear algebra on above

Ok let’s Have The Fun!

I encourage you to run following code into Jupyter Notebook that is running With Julia-Kernel.

π

typeof(π)# it will return irrational. Beacuse pi is irrational Number? 😉

Let’ Hack Julia’s Type System on Much deeper level!(Because it is much more than classes)

What else we need to know about it?

Define new Parametric Type in Julia:

type vector_3d{T<:Integer}
x::T
,y::T
end

# can we call x any as Data-Members as like as C++ Data-Members?

type_call = vector_3d{25,25} # this is how we call it.

Let’s Just make Types more interesting: (and immutable)

immutable GF{P,T<:Integer} <: Number
data::T
function GF(x::Integer)
return new(mod(x, P))
end
end

Deep Learning and Machine Learning in Julia:

In the real eye Julia is developed to write “Mathematical Functions” by just using Native Language Syntax. It is more like if you want to do linear regression rather than installing a New_library and calling inbuilt Linear function of that library those could be written in C, C++ or Fortran may be or More or less Optimized Cython-Python Magic. But Julia responsibly provides static inbuilt and Really fast code methods to write your Own linear regression as easy as Python and as Fast as C++/Fortran.

Available Machine-Larning Packages in Julia:

Scikit-Learn in Julia:

ScikitLearn.jl implements the popular scikit-learn interface and algorithms in Julia. It supports both models from the Julia ecosystem and those of the scikit-learn library (via PyCall.jl).

https://github.com/cstjean/ScikitLearn.jl

Text Analysis in Julia:

The basic unit of text analysis is a document. The TextAnalysis package allows one to work with documents stored in a variety of formats:

  • FileDocument: A document represented using a plain text file on disk
  • StringDocument: A document represented using a UTF8 String stored in RAM
  • TokenDocument: A document represented as a sequence of UTF8 tokens
  • NGramDocument: A document represented as a bag of n-grams, which are UTF8 n-grams that map to counts

https://github.com/JuliaText/TextAnalysis.jl

Machine-Learning Package with name Machine_learning:

The MachineLearning package represents the very beginnings of an attempt to consolidate common machine learning algorithms written in pure Julia and presenting a consistent API. Initially, the package will be targeted towards the machine learning practitioner, working with a dataset that fits in memory on a single machine. Longer term, I hope this will both target much larger datasets and be valuable for state of the art machine learning research as well.

https://github.com/benhamner/MachineLearning.jl

Deep Learning in Julia:

Mocha is a Deep Learning framework for Julia, inspired by the C++ framework Caffe. Efficient implementations of general stochastic gradient solvers and common layers in Mocha can be used to train deep / shallow (convolutional) neural networks, with (optional) unsupervised pre-training via (stacked) auto-encoders. Some highlights:

https://github.com/pluskid/Mocha.jl

Deep Learning with Automatic Differentiation:(What is automatic Differentiation?)

Knet (pronounced “kay-net”) is the Koç University deep learning framework implemented in Julia by Deniz Yuret and collaborators. It supports GPU operation and automatic differentiation using dynamic computational graphs for models defined in plain Julia. This document is a tutorial introduction to Knet. Check out the full documentation and Examples for more information. If you need help or would like to request a feature, please consider joining the knet-users mailing list. If you find a bug, please open a GitHub issue. If you would like to contribute to Knet development, check out the knet-dev mailing list and Tips for developers.

https://github.com/denizyuret/Knet.jl

More resources on Julia Programing:

http://online.kitp.ucsb.edu/online/transturb17/gibson/

https://julialang.org/blog/

Feel free to clap and Have fun with Julia. Stay connected.

Programmer’s or Entrepreneur’s life Guide(How to live!)

I just completed Audio version ofSoft Skills: The software developer’s life manualand same after completing of new-skills/books/courses I go with a blog-post, Because I always feel that you can never observer whole book by reading it once and you always need to re-learn things over time because sometimes to learn really new things, you also have to forget  some of  the stuff to learn some of new things as well, And I guess that is one of the most important thing I learned over last few years after my College. So writing a Blog-Post is one of thing I do to Enhance/Validate/Preserve new learnings.

When I first saw John Z. Sonmez on Youtube My first thought was like ” He is so fit, Is he really Software-Engineer” 🙂 How wrong I was.

You can get connected with John here.

 

Ok here comes the Validation of Knowledge about the book mentioned above. You can also use this blog-post to get Excited about book and add it to your Reading List. I am sure you will feel that here is something so common but fresh-about this book, As I am feeling right now that may be I will always read/listen this book at-least once a year.

Be a Specialist: Special Skill is like Brahmanda Astra.

it really does not matter how much you are curios about new technologies but there must be one skill that you will be so good that you can represent yourself at top 1% in your Area/Geo-location/Company/Industry and make sure while doing good amount of work using that skill your produce more Quality than quantity. Alongside that special-skill you must be knowing about sister technologies of that particular skill. There is one more another important thing to remember and that is “Don’t get religious about any particular Technology. “

Learn to Sell yourself High Enough.

Selling yourself only means “Be so great so that nobody can ignore you” but that also does not mean you must write code and build great-softwares by sitting 18-20 hours for day in your basement. You must have off-line as well as on-line good circle of Professional and Merchants in your life. You must have good amount of blog post about the skills you you have learned over time alongside the skills you are learning now due to that you would be so connected with the outside world and most of the people would be knowing that what you really are doing in life.

 Salary Negotiation: It’s always good to negotiate because your Employer know that “It’s good to bargain”

When it comes to salary negotiation it is true that you must be able to know how much you are worth as well as how much more money you need except from your living Expenses. No doubt if you have reached at the level of Salary-Negotiation that means you have already know how much Employer could offer but it never hurts to take that “Moment into bit more negotiation”, Now may be A question could be arise in your mind and that is How really I have to do that? Believe me I am not the guy who can explain it so good as John has descried in the book. So read the book. If you are fan of Audio-Books then Let me know, I will send you Audio version of that book But you will only able to get it free if this is your first book on Audible.com

Investment and Saving: Let your money work for you and you just be sit around to see it is growing. 🙂

So It is really easy to do actually, You just need to understand How much you are able to save and how much you really can do with that saving, Confusing? Money sitting in Bank has no value until or unless you are able to make more of that than Banks do make, and how is that thing possible? There are Wide range of options one can go for like Options-Trading, Futures-Trading, Real State Investment, Making/Investing in one of your own Software Product and many other things. 🙂

For more information about Stocks and Futures please follow the following Articles:

View story at Medium.com

Body and Mental Fitness: The way you look, The way your brain does lot of work for you. 😉

 

There must be really good work-out routine we have to follow/do unless or until you really are not interested growing your brain over the time, Yes that is true only because you have Pump your blood almost everyday and you also have to burn fat, So start running and going to Gym because it is the only way you can not just make your body healthy but also sharp your mind over time.

Diet and Nutrition: You are what you consume.

There is nothing much to say about Diet and eating habits, But make sure you are eating less calories whole day because you have to sit for the most of the day. 🙂

Power of Persistence: Hang in there!

Things work as much as you are staying there for things to make work. Sometimes it takes some time for the results to show up but at some level you have to wait more and let the things fall into the place.

Special One: How money Works.

I never expected John will also talk about great things about Money and Capitalism as well, To understand this you  must have to learn so much stuff and I might not be the person to tell you on this short post that how money works and how you should take it into account. Simple thing about Money is Banks are creating lots of vitalized clusters to generate/initiate the flow of money among different countries and different inside different industries.

Asynchronous recipes in Python

“Concurrency” is not “Parallelism” May be it’s better. If you will not work with DataScience, DataProcessing, Machine-Learning and other operations which are CPU-Intensive you probably will found that you don’t need parallelism but you need concurrency more!

  1. A Simple Example is Training a machine learning model is CPU intensive or You can use GPU.
  2. To Make various Predictions from one model based on many Different Input-Parameters to find out best result You need Concurrency!

There are so Many ways one can hack into Python stuff and do cool Stuff either it is CPU intensive or just a task to do stuff that is good/bad/Better/Best for one user to communicate. One thing you have to believe that Python Does support Multiprocessing as well as Multi-threading

but for various reasons when you are doing CPU intensive Tasks you have to Stay away from using Threading operations in Python. Use Numpy, Cython,Jython or anything you feel, Write C++ code and glue it with Python

The number of threads will usually be equivalent to the number of cores you have. If you have hyperthreading on your processor, than you will be able to double the number of threads used.

Above image is just one Example to understand what actually we are doing. We are processing Chunks and Chuncks of Data. Now the real common scenario is If you are using I/O bound tasks use Threads in Python if you are using CPU bound tasks use Processes in Python.  I have worked with various Python Projects where Performance was issue at some level so at that time I always went to other things like Numpy, Pandas, Cyhton or numba but not Plain-Python.

Let’s come to the point and Point is What are those Recipes I can use:

Using concurrent.futures(futures module is also back-ported into Python2.x):

Suppose you have to call multiple URLs at same time using same Method. That is what actually Concurrency is, Apply same method different operations, We can do it either using ThreadPool or ProcessPool.

# Using Process Pool
from concurrent.futures import ProcessPoolExecutor,as_completed
def health_check1(urls_list):
pool = ProcessPoolExecutor(len(urls_list))
futures = [pool.submit(requests.get,url,verify=False) for url in final_url]
results = [r.result() for r in as_completed(futures)] # when all operations done
return results # a Python list of all results, Here you can also use Numpy as well

Using ThreadPool it is also not different:

# Using Thread Pool
from concurrent.futures import ThreadPoolExecutor,as_completed</code>

def just_func(urls_list):
pool = ThreadPoolExecutor(len(urls_list))
futures = [pool.submit(requests.get,url,verify=False) for url in urls_list]
results = [r.result() for r in as_completed(futures)] # when all operations done
return results # a Python list of all results, Here you can also use Numpy as well

In the above code ‘url_list’ is just list of tasks which are similar and can be processed using same kind of functions.

On the other-side using it with with as context manager is also not different. In this Example I will Use ProcessPoolexecutor’s inbuilt map function.

def just_func(url_list):
   with concurrent.futures.ProcessPoolExecutor(max_workers=len(final_url)) as executor:
        result = executor.map(get_response,final_url)
    return [i for i in result]

Using multiprocessing: (Multiprocessing is also Python-library that can be used for Asynchronous behavior of your code.)

*in Multiprocessing the difference between map and apply_async is only that Map returns results as task list is passed to it on the other-hand apply_async returns results based on results those returned by function.

# Function that run multiple tasks
def get_response(url):
“””returns response for URL ”””
    response = requests.get((url),verify=False)
   return response.text

Now above function is simple enough that is getting one URL and returning response but if have to pass multiple URLs but I want that get request to each URL should be fired at same time then That would be Asynchronous process not multiprocessing because in Multiprocessing Threads/Processes needs to communicate with each other but on the other hand in case of Asynchrounous threads don’t communicate(in Python because Python uses Process based multiprocessing not Thread Based although you can do thread-based multiprocessing in Python but then you are on your OWN 😀 😛 Hail GIL (Mogambo/Hitler)).

So above function will be like this as usual:

from multiprocessing import Pool
pool = Pool(processes=20)
resp_pool = pool.map(get_response,tasks)
URL_list = []
resp_pool = _pool.map(get_response,tasks)
pool.terminate()
pool.join()

Although This is an interesting link one can watch while going into Multiprocessing in Python using Multiprocessing: It is Process-Bases Parallelism.
http://sebastianraschka.com/Articles/2014_multiprocessing.html

Using Gevent: Gevent is a concurrency library based around libev. It provides a clean API for a variety of concurrency and network related tasks.

import gevent
import random

def task(pid):
"""
Some non-deterministic task
"""
    gevent.sleep(random.randint(0,2)*0.001)
    print('Task %s done' % pid)

def asynchronous():
    threads = [gevent.spawn(task, i) for i in xrange(10)]
    gevent.joinall(threads)

    print('Asynchronous:')
asynchronous()

If you have to Call Asynchronously but want to return results in Synchronous Fashion:

import gevent.monkey
gevent.monkey.patch_socket()

import gevent
import urllib2
import simplejson as json

def fetch(pid):
    response = urllib2.urlopen('http://json-time.appspot.com/time.json')
    result = response.read()
    json_result = json.loads(result)
    datetime = json_result['datetime']

    print('Process %s: %s' % (pid, datetime))
    return json_result['datetime']

def asynchronous():
    threads = []
    for i in range(1,10):
        threads.append(gevent.spawn(fetch, i))
    gevent.joinall(threads)

print('Asynchronous:')
asynchronous()

Assigning Jobs in Queue:

import gevent
from gevent.queue import Queue</code>

tasks = Queue()

def worker(n):
    while not tasks.empty():
        task = tasks.get()
        print('Worker %s got task %s' % (n, task))
   gevent.sleep(1)

print('Quitting time!')

def boss():
   for i in xrange(1,25):
      tasks.put_nowait(i)

    gevent.spawn(boss).join()

   gevent.joinall([
gevent.spawn(worker, 'steve'),
gevent.spawn(worker, 'john'),
gevent.spawn(worker, 'nancy'),
])

When you have to manage Different Groups of Asynchronous Tasks:

import gevent
from gevent.pool import Group</code>

def talk(msg):
    for i in xrange(3):
        print(msg)

g1 = gevent.spawn(talk, 'bar')
g2 = gevent.spawn(talk, 'foo')
g3 = gevent.spawn(talk, 'fizz')

group = Group()
group.add(g1)
group.add(g2)
group.join()

group.add(g3)
group.join()

Same As multiprocessing Library you can also use Pool to map various operations:

import gevent
from gevent.pool import Pool</code>

pool = Pool(2)

def hello_from(n):
    print('Size of pool %s' % len(pool))

pool.map(hello_from, xrange(3))

Using Asyncio:

Now let’s talk about concurrency Again! There is already lot of automation is going inside asyncio or Gevent but as programmer we have to understand how we need to break a “One large task into small chuncks of Subtasks so when we will write code we will be able to understand which tasks can work independently.

import time
import asyncio

start = time.time()

def tic():
    return 'at %1.1f seconds' % (time.time() - start)

async def gr1():
# Busy waits for a second, but we don't want to stick around...
    print('gr1 started work: {}'.format(tic()))
    await asyncio.sleep(2)
    print('gr1 ended work: {}'.format(tic()))

async def gr2():
# Busy waits for a second, but we don't want to stick around...
    print('gr2 started work: {}'.format(tic()))
    await asyncio.sleep(2)
    print('gr2 Ended work: {}'.format(tic()))

async def gr3():
    print("Let's do some stuff while the coroutines are blocked, {}".format(tic()))
    await asyncio.sleep(1)
    print("Done!")

ioloop = asyncio.get_event_loop()
tasks = [
ioloop.create_task(gr1()),
ioloop.create_task(gr2()),
ioloop.create_task(gr3())
]
ioloop.run_until_complete(asyncio.wait(tasks))
ioloop.close()

Now in the above code gr1 and gr2 are somehow taking some time to return anything it could any kind of i/o operation so what we can do here is go to the gr3 in using the event_loop and event_loop will run until all three tasks are not completed.

Please have a closer look at await keyword in the above code. It is one of the most important step where you can assume interpreter is shifting from one task to another or you can call it pause for function. If you have worked with yield or yield from in Python2 and Python3 you would be able to understand that this is stateless step for the code.

There is on more library which is aiohttp that is being used to handle blocking Http requests with asyncio.

import time
import asyncio
import aiohttp

URL = 'https://api.github.com/events'
MAX_CLIENTS = 3

async def fetch_async(pid):
    print('Fetch async process {} started'.format(pid))
    start = time.time()
    response = await aiohttp.request('GET', URL)
    return response

async def asynchronous():
    start = time.time()
   tasks = [asyncio.ensure_future(fetch_async(i)) for i in range(1, MAX_CLIENTS +1)]
   await asyncio.wait(tasks)
    print("Process took: {:.2f} seconds".format(time.time() - start))

print('Asynchronous:')
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(asynchronous())
ioloop.close()

In all the above Examples we have just Scratched the world of concurrency but in real there would be much more to look into because real world problems are more complex and intensive. There are various other options in asyncio like handling exceptions with-in futures, creating future wrappers for normal tasks,Applying timeouts if task is taking more than required time and doing something else instead.

There is lot of inspiration I got while learning about concurrent programming in Python from the following Sources:

https://hackernoon.com/asyncio-for-the-working-python-developer-5c468e6e2e8e
http://www.gevent.org/
https://www.binpress.com/tutorial/simple-python-parallelism/121
http://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.html

Running Multiprocessing in Flask App(Let’s Spawn) Hell Yeah

Ok It was going to be long time but Finally yeah Finally Able to do Process based multiprocessing in Python and even on Flask. 🙂 oh yeah! There are various recipes for Multiprocessing in this python but here you can only Enjoy with Flask.

:D


from multiprocessing import Pool
from flask import Flask
from flask import jsonify
import ast
import pandas as pd
import requests

app = Flask(__name__)
_pool = None

# Function that run multiple tasks
def get_response(x):
"""returns response for URL list"""
m = requests.get((x),verify=False)
return m.text

@app.route('/call-me/')
def health_check():
"""returns pandas dataframe into HTML for health-check Services"""
resp_pool = _pool.map(get_response,tasks)
table_frame= pd.DataFrame([ast.literal_eval(resp) for resp in resp_pool])
return table_frame.to_html()

if __name__=='__main__':
_pool = Pool(processes=12) # this is important part- We
try:
# insert production server deployment code
app.run(use_reloader=True)
except KeyboardInterrupt:
_pool.close()
_pool.join()

 

Everything you should know about Quantitative Finance(Almost Everything ;)

Quant trading is intellectually very satisfying because it draws heavily from fields as diverse as computer science, statistics, math, psychology, economics, business, operations, research, and history.
Quant trading combines programming and trading. Programming is the fastest way to take an idea and turn it into reality (compare it to writing a book, publishing, etc or to architecture a building, getting zoning permits,doing construction, etc). Trading is the most direct approach to making money (compare it with starting the next facebook, programming, hiring, advertising, etc or to creating a retail chain, hiring, waiting for money to flow in to reinvest, etc). Combining programming and trading,  you have the most direct path from an idea in your brain to cash but never forget that all you have to do is minimize Risk.

Quant trading is intellectually very satisfying because it draws heavily from fields as diverse as computer science, statistics, math, psychology, economics, business, operations research, and history.

Trading and Trading Strategies:

There are many words those sound complicated when you read or listen about Trading things for example on of those is Statistical Arbitrage Trading. It looks complicated but in real manner it is another form of Pair’s Trading. Pair’s trading is often considered as selling/buying two inter-related stocks/commodities/currencies/futures. Trading strategy is set of calculations those a Trader/Quant/Person believes that will work-out in near or long future in trading markets. There are various methods of trading like Fully-Automated or Semi-Automated. To write a Trading strategy one does not need to know programming or advanced mathematics but on the other hand one need to understand simple concepts of Statistical methods. On the other hand a degree in Advances degree in Mathematics or similar also does not assure that one can write “Profitable” Trading strategies. I believe it is more the  matter of Past experience in Trading as well.

How Easy it is to earn from Trading:

When we talk about programming languages often one question is asked, ‘How fast it is?’ and always counter question is ‘Define Fast’. I would like to implement same phenomenon here and that is ‘define easy’ or we can sat that Earning from Trading is EASY but we must not consider easy as ‘Instant’. Most of the times people or brokerage firms  are more interested in intra-day trading because big volume is there and that means more buy and sell opportunities.

Borrow Money While Trading:

Good thing about Trading is you have to small amount of Money and you can always get leverage. Leverage is kind you can lend from trading firms for trading so if you will get profit it will also be shared with Firm and on the other hand in the case of loss it will be all yours. Good trading Strategy is considered as one having good returns, better sharp Ratio, small draw-downs, and   uses as low leverage as possible, Suppose you got very good tip from anywhere about one really interesting trading investment that you are pretty sure will work even in that case you should use as less as possible. it is dangerous to over-leverage in pursuit of overnight riches.

How much time is required for self running Quantitative Business:

Time is really dependent on the type of Trading strategies you are using. Some trading strategies or Machine-learning models need to be trained few minutes  before market opens  as well as some models need to be trained after closing the market. Sometimes one has to come up with trading strategies those train models once a week and runs the Buy/Sell calls once a Month.

How Hard it is to Create Trading Strategies:

Creating Trading strategies are as hard as doing calculus and derivative or as easy as finding mean or median from a given set of data. You might talk about various things while creating Trading Strategies but simpler one is think more and more about Quantitative Techniques. Don’t look at the preciseness, perfect or most accurate results you want to achieve but look for the valuable information or quantify those valuable things you need to know about your Trading model.

Importance of Programming in Quantitative Finance:

Developing a Trading Strategy is taste of mind. One can develop various trading strategies by just looking at the things and writing a daily schedule like generate a Buy call at this time and Generate a Sell call at this time and that would also workout but real reason behind all this is you want to create a trading-Bot/Robot that does it automatically so you could observe it as external factor and  look into various things those can be tuned so converting a Trading Strategy into Program/code/algorithm is one of the most important task we should persuade.

A Dollar-neutral Portfolio:

The market value of the long positions equals the market value of the short positions.

Market-Neutral Portfolio:

The beta of the portfolio with respect to a market index is close to zero, where beta measures the ratio between
the expected returns of the portfolio and the expected returns of the market index) require twice the capital or leverage of a long- or short-only portfolio.

 

Directional Strategy – That only does one thing either buy or sell.
Bi-directional Strategy – That does both buy and sell.

 

Survivorship Bias Free Data and it’s importance:

One can use a database as Back-test that is not survivorship Bias free for intra-day trading as well because in most of the cases a good company does not fall in one day and same happens with rise of company.

How holding period does effect you Trading Strategies:

I was doing trading strategies based on various factors, Using machine-learning I was predicting if tomorrow’s close price is higher than today’s then buy otherwise sell the stock. it is great Practice to do but in real we have to understand one thing carefully and that is I am holding a stock for whole day, Anything could go wrong in whole day so what I had to do is shorter the holding period of the stock, That means for less time I will hold that particular stock less chances will be there for anything Going wrong and less Risk. 🙂

Sharp-Ratio(Most important thing):

Sharp Ratio defines how consistent your returns are! Think of sharp-ratio as independent from Benchmark if you want to create Strategy with Good returns and High Sharpe Ratio that means your approximation or understanding of Market should come up with various rules of Generalization. Your Sharpe ration should be more than 1(one), Strategy returns could be less or more than  benchmark returns. Sharp-Ratio is also considered as information-Ratio and formula to calculate sharp-Ratio is as follows:
Information Ratio(Sharpe-Ratio) = Average of Excess Returns/Standard Deviation of Excess Returns
Excess Returns = portfolio returns- benchmark returns
As a rule of thumb, any strategy that has a Sharpe ratio of less than 1 is not suitable as a stand-alone strategy. For a strategy that achieves profitability almost every month, its (annualized) Sharpe
ratio is typically greater than 2. For a strategy that is profitable almost every day, its Sharpe ratio is usually greater than 3.

One important thing you mush be knowing that Benchmarks varies according to the Market/Exchange you are working with.

Sharpe-Ratio Vs Returns:

If the Sharpe ratio is such a nice performance measure across different strategies, you may wonder why it is not quoted more often instead of returns. A higher Sharpe ratio will actually allow you to make more profits in the end, since it allows you to trade at a higher leverage. It is the leveraged return that matters in the end, not the nominal return of a trading strategy.

Why and How Draw-Down is bad:

A strategy suffers a draw-down whenever it has lost money recently.
A draw-down at a given time t is defined as the difference between the current equity value (assuming no redemption or cash infusion) of the portfolio and the global maximum of the equity curve occurring on or before time t.

You must know draw-down of strategy before using it!

Length and depth of your Draw down:

Your draw-down’s length defines how long it would take to recover the market and depth defines how much you can loose, but those results are based on your back-testing in real you have ti understand things in better way because in the real trading strategies Draw-Downs could be very less or more than benchmark results.
What is Slippage— ?

This delay can cause a “slippage,” the difference between the price that triggers the order and the execution price.

How Will Transaction Costs Affect the Strategy?
Transaction cost is brokerage amount or something that costs when you buy/sell any stock. Now as we understand that less the hold period will be more profit you can make or at-least less risk you will be having. One thing you must never forget and that is minimize the risk that is all behind the strategies and Algorithmic Trading. Every time a strategy buys and sells a security, it incurs a transaction cost. The more frequent it trades, the larger the impact of transaction costs will be on the profitability of the strategy. These transaction costs are not just due to commission fees charged by
the broker. There will also be the cost of liquidity—when you buy and sell securities at their market prices, you are paying the bid-ask spread. If you buy and sell securities using limit orders, however,
you avoid the liquidity costs but incur opportunity costs. This is because your limit orders may not be executed, and therefore you may miss out on the potential profits of your trade. Also, when you buy or sell a large chunk of securities, you will not be able to complete the transaction without impacting the prices at which this transaction is done. This effect on the market prices due to your own order is called market impact, and it can contribute to a large part of the total transaction cost when the security is not very liquid.

Why survivorship bias should not be there?

This is especially true if the strategy has a “value” bent; that is, it tends to buy stocks that are cheap. Some stocks were cheap because the companies were going bankrupt shortly. So if your strategy includes only those cases when the stocks were very cheap but eventually survived (and maybe prospered) and neglects those cases where the stocks finally did get de-listed, the backtest performance will, of course, be much better than what a trader would actually have suffered at that time.

Data-Snooping Bias?(Model Over-fitting)
If you build a trading strategy that has 100 parameters, it is very likely that you can optimize those parameters in such a way that the historical performance will look fantastic. It is also very likely that the future performance of this strategy will look nothing like its historical performance and will turn out to be very poor. By having so many parameters, you are probably fitting the model to historical accidents in the past that will not repeat themselves in
future. Actually, this so-called data-snooping bias is very hard to avoid even if you have just one or two parameters (such as entry and exit thresholds).

Important Questions to ask yourself:

1. How much time do you have for baby-sitting your trading programs?
2. How good a programmer are you?
3. How much capital do you have?
4. Is your goal to earn steady monthly income or to strive for a large, long-term capital gain?

Important questions you must ask yourself before using a Trading Strategy:
1. Does it outperform a benchmark?
2. Does it have a high enough Sharpe ratio?
3. Does it have a small enough drawdown and short enough draw-
down duration?
4. Does the backtest suffer from survivorship bias?
5. Does the strategy lose steam in recent years compared to its ear-
lier years?
5. Does the strategy have its own “niche” that protects it from intense  competition from large institutional money managers?

Sharp-Ratio and drop-downs(length and duration):
Quantitative traders use a good variety of performance measures. Which set of numbers to use is sometimes a matter of personal preference, but with ease of comparisons across different strategies and traders in mind, I would argue that the Sharpe ratio and draw-downs are the two most important. Notice that I did not include average annualized returns, the measure most commonly quoted by investors, because if you use this measure, you have to tell people a number
of things about what denominator you use to calculate returns. For example, in a long-short strategy, did you use just one side of capital or both sides in the denominator? Is the return a leveraged one (the
denominator is based on account equity), or is it leveraged (the denominator is based on market value of the portfolio)? If the equity or market value changes daily, do you use a moving average as the denominator, or just the value at the end of each day or each month? Most (but not all) of these problems associated with comparing re-turns can be avoided by quoting Sharpe ratio and draw-down instead as the standard performance measures.

Interesting things about Back-Testing:

Back-testing is the process of creating the historical trades given the historical information available at that time, and then finding out what the subsequent performance of those trades is. This process seems easy given that the trades were made using a computer algorithm in our case, but there are numerous ways in which it can go wrong. Usually, an erroneous back-test would produce a historical performance that is better than what we would have obtained in actual trading. We have already seen how survivorship bias in the data used for back-testing can result in inflated performance.

Importance of Sample Size (How much historical data you need?):
The most basic safeguard against data-snooping bias is to ensure that you have a sufficient amount of back-test data relative to the number of free parameters you want to optimize. As a rule of thumb, let’s assume that the number of data points needed for optimizing your parameters is equal to 252 times the number of free parameters your model has. So, for example, let’s assume you have a daily trading model with three parameters. Then you should
have at least three years’ worth of back-test data with daily prices.

However, if you have a three-parameter trading model that updates positions every minute, then you should have at least 252/390 year, or about seven months, of one-minute back-test data. (Note that if
you have a daily trading model, then even if you have seven months of minute-by-minute data points, effectively you only have about 7 × 21 = 147 data points, far from sufficient for testing a three parameter model.)

Training-Set and Test-Set:

It is very simple concept. You have to split data into two portions one would be training set for your model to learn and other would be Test-Set.

What is Profit cap?:

Profit cap is limit that at what amount you want your strategy to exit. In real achieving a profit cap is ultimate goal for the strategy, A greedy strategy without profit cap and Stop-loss could destroy all of your liquidity.
Parameter-less model:

This is like self sustaining model that does all the stuff by itself, that means you need to make sure that your model  is safe and secure and all the parameters like profit-cap is being calculated by itself.

The advantage of a parameterless trading model is that it minimizes the danger of over-fitting the model to multiple input parameters (the so-called “data-snooping bias”). So the back-test performance should be much closer to the actual forward performance. (Note that parameter optimization does not necessarily mean picking one best set of parameters that give the best back-test performance. Often, it is better to make a trading decision based on some kind of averages over different sets of parameters.)

A Simple understanding of Back-testing:

Backtesting is about conducting a realistic historical simulation of the performance of a strategy. The hope is that the future performance of the strategy will resemble its past performance, though as your investment manager will never tire of telling you, this is by no means guaranteed!
There are many nuts and bolts involved in creating a realistic historical back-test and in reducing the divergence of the future Backtesting.

Things to take care in Back-Testing:

Data: Split/dividend adjustments, noise in daily high/low, and survivorship bias.
Performance measurement: Annualized Sharpe ratio and maximum draw-down.
Look-ahead bias: Using unobtainable future information for past trading decisions.
Data-snooping bias: Using too many parameters to fit historical data, and avoiding it using large enough sample, out-of-sample testing, and sensitivity analysis.
Transaction cost: Impact of transaction costs on performance.
Strategy refinement: Common ways to make small variations on the strategy to optimize performance.

 

Importance of Speed in Algorithmic Trading:

There are various things included when you talk about HFT and speed. It does matter that which part of your Trading algorithm takes much more time for execution. Think of this as 90/10 rule in software development. Optimize that 10% portion of your code that Takes 90% time. If your Trading strategy is written in Python that means it could be slow on various portions so it’s better to use C or C++ for such purpose but on the other hand you can also use Cython which is really fast in the case of development as well as in the case of Execution of code. On the other hand your internet connection should be fast as well so you would be able to get data really fast and make decisions based on that data.

Let’s again talk about importance of Programming:

You can use various available custom Platforms for trading as well as you can also create custom ones that uses various back-tests, Different Trading Exchanges(Using APIs),Different Machine-learning models as well as different Platforms and Programming Languages.
How to Decrease Brokerage Cost using Programming?:
In order to minimize market impact cost, you should limit the size (number of shares) of your orders based on the liquidity of the stock. One common measure of liquidity is the average daily volume (it is your choice what lookback period you want to average over).
As a rule of thumb, each order should not exceed 1 percent of the average daily volume. As an independent trader, you may think that it is not easy to reach this 1 percent threshold, and you would be right when the stock in question is a large-cap stock belonging to the S&P 500. However, you may be surprised by the low liquidity of
some small-cap stocks out there.

 

Paper Trading and Re-creating Strategies:(Testing in Real Market)
The moment you start paper trading you will realize that there is a glaring look-ahead bias in your strategy—there may just be no way you could have obtained some crucial piece of data before you enter an order! If this happens, it is “back to the drawing board.” You should be able run your ATS, execute paper trades, and then compare the paper trades and profit and loss (P&L) with the theoretical ones generated by your backtest program using the latest data. If the difference is not due to transaction costs (including an expected delay in execution for the paper trades), then your software likely has bugs.

Another benefit of paper trading is that it gives you better intuitive understanding of your strategy, including the volatility of its P&L, the typical amount of capital utilized, the number of trades per day, and the various operational difficulties including data issues. Even though you can theoretically check out most of these features
of your strategy in a back-test, one will usually gain intuition only if
one faces them on a daily, ongoing basis. Back-testing also won’t reveal the operational difficulties, such as how fast you can download all the needed data before the market opens each day and how you
can optimize your operational procedures in actual execution.

Psychology and Trading:

This is one of the most important concept you must be knowing. Trading is real money and that real money could make you really mad and in lots of ways. I am again pointing this thing out which is Algorithmic Trading is all about Minimizing your Risk not to get Really rich instantly. Yes you can create Strategies with high Sharpe ratio and sell it to firms like JP Morgan or other big Banks then you will be rich very quickly. 🙂

More or less it’s not just the Trading-Strategy you create but it’s your mind and experiences as well those help you to grow as well as your Capital.

How we can implement RISK-Management Using Programming?:

Calculating the Kelly criterion is relatively simple and relies on two basic components: your trading strategy’s win percentage probability and its win to loss ratio.

The win percentage probability is the probability that a trade will have a positive return. The win to loss ratio is equal to your total trading profits divided by your total trading losses.

These will help you arrive at a number called the Kelly percentage. This gives you a guide to what percentage of your trading account is the maximum amount you should risk on any given trade.

The formula for the Kelly percentage looks like this:

Kelly % = W – [(1 – W) / R]

  • Kelly % = Kelly percentage
  • W = Win percentage probability (total number of winning trades/total number of trades)
  • R = Win to loss ratio (total amount gained by winning trades/total amount lost by losing trades)

References:

Max Dama:http://www.decal.org/file/2945

Quantitative Trading by Ernie P Chan: http://www.amazon.in/Quantitative-Trading-Build-Algorithmic-Business/dp/0470284889

Google Search: http://en.tradimo.com/learn/money-management/kelly-criterion/

Important Julia Packages

  1. Julia Ipython

Julia is able to run very well on you Ipython notebook Environment. After all, All you have to do is Data-Science and Machine-Learning. 🙂

julia

1.1 Open Julia Prompt(At Ubuntu it works like typing ‘julia’ command in your Terminal)

1.2 run command > Pkg.add(“IJulia”) # it will do almost all the work.

2. DataFrames: Whenever you have to read lot of files in Excel-Style Julia DataFrames Package is good to go.

Pkg.add("DataFrames")

3. Arduino:

A Julia Package for interacting with Arduino.

https://github.com/rennis250/Arduino.jl

4. Neural Network Implementation of Julia

https://github.com/compressed/BackpropNeuralNet.jl

5. Visualizing and Plotting in Julia:

https://github.com/bokeh/Bokeh.jl

6. Reading and writing CSV files in Julia

https://github.com/JuliaData/CSV.jl

7. DataClusting in Julia:

https://github.com/JuliaStats/Clustering.jl

For more Large number of Packages, Please refer following link:

http://pkg.julialang.org/

Note*: You can also run most of the Shell commands in Julia environment as well. 🙂

%d bloggers like this: