Class09 Answer:

Generate Features

When I work with stock market data, I generate features from slopes of price moving averages.

When I work with Forex data, for each observation, I generate features from lagging prices.

For this lab, the script I used to create features for each observation is listed below:


# genf.py

# This script should generate features from timestamps and prices.
import pandas as pd
import numpy  as np
# I should specify duration I hold a pair after I act on a prediction:
duration_i = 12 # 12 means 12 * 5 which is 1 hour which is 3600 seconds
pairs_l    = ['AUDUSD','EURUSD','GBPUSD','USDCAD','USDJPY']
for pair_s in pairs_l:
  print(pair_s+' Busy...')
  g0_df = pd.read_csv("../csv/"+pair_s+".csv", names=['ts','cp'])
  # I should compute dependent variable, piplead:
  g0_df['piplead'] = (10000.0*(g0_df.cp.shift(-duration_i) - g0_df.cp) / g0_df.cp).fillna(0)
  # I should compute piplags 1,2,4,8
  g0_df['piplag1'] = (10000.0*(g0_df.cp - g0_df.cp.shift(duration_i*1)) / g0_df.cp).fillna(0)
  g0_df['piplag2'] = (10000.0*(g0_df.cp - g0_df.cp.shift(duration_i*2)) / g0_df.cp).fillna(0)
  g0_df['piplag3'] = (10000.0*(g0_df.cp - g0_df.cp.shift(duration_i*3)) / g0_df.cp).fillna(0)
  g0_df['piplag4'] = (10000.0*(g0_df.cp - g0_df.cp.shift(duration_i*4)) / g0_df.cp).fillna(0)
  g0_df['piplag5'] = (10000.0*(g0_df.cp - g0_df.cp.shift(duration_i*5)) / g0_df.cp).fillna(0)
  # I should write to CSV file to be used later:
  g0_df.to_csv("../csv/feat"+pair_s+".csv", float_format='%4.4f', index=False)
  
'bye'

I ran the above script and it needed about one minute to finish.

The resulting CSV files are listed below:


fx411@ub16aug:~/fx411/script$ 
fx411@ub16aug:~/fx411/script$ 
fx411@ub16aug:~/fx411/script$ python genf.py 
AUDUSD Busy...
EURUSD Busy...
GBPUSD Busy...
USDCAD Busy...
USDJPY Busy...
fx411@ub16aug:~/fx411/script$ 
fx411@ub16aug:~/fx411/script$ ll ../csv/feat*csv
-rw-rw-r-- 1 fx411 fx411 35637725 Nov  8 18:36 ../csv/featAUDUSD.csv
-rw-rw-r-- 1 fx411 fx411 35375195 Nov  8 18:36 ../csv/featEURUSD.csv
-rw-rw-r-- 1 fx411 fx411 35275476 Nov  8 18:36 ../csv/featGBPUSD.csv
-rw-rw-r-- 1 fx411 fx411 35099775 Nov  8 18:36 ../csv/featUSDCAD.csv
-rw-rw-r-- 1 fx411 fx411 36256235 Nov  8 18:37 ../csv/featUSDJPY.csv
fx411@ub16aug:~/fx411/script$ 
fx411@ub16aug:~/fx411/script$ 
fx411@ub16aug:~/fx411/script$


fx411@ub16aug:~/fx411/script$
fx411@ub16aug:~/fx411/script$
fx411@ub16aug:~/fx411/script$ tail ../csv/featAUDUSD.csv
1477984200,0.7607,0.0000,-1.8666,1.8798,-4.4694,2.4713,-1.4854
1477984500,0.7609,0.0000,-0.7097,4.6000,-2.3131,5.8748,-2.1554
1477984800,0.7606,0.0000,-3.7468,2.6031,-5.8766,2.8923,-4.7723
1477985100,0.7604,0.0000,-5.3392,-3.0115,-6.7594,-0.2762,-5.3786
1477985400,0.7603,0.0000,-6.9050,-4.1956,-7.6546,-3.8142,-5.2215
1477985700,0.7605,0.0000,-5.5228,0.0000,-5.5754,-2.5116,-4.2210
1477986000,0.7605,0.0000,-7.7319,-0.6180,-5.1809,-2.3538,-2.4721
1477986300,0.7607,0.0000,-6.3629,0.2498,-1.6302,-0.5522,1.0386
1477986600,0.7606,0.0000,-5.9951,-0.7231,-1.8537,-1.1175,0.8283
1477986900,0.7607,0.0000,-5.9422,-0.8019,-1.9851,-0.6573,3.0237
fx411@ub16aug:~/fx411/script$
fx411@ub16aug:~/fx411/script$


fx411@ub16aug:~/fx411/script$ 
fx411@ub16aug:~/fx411/script$ date -u --date=@1477986900
Tue Nov  1 07:55:00 UTC 2016
fx411@ub16aug:~/fx411/script$ 

Class09 Lab


learn4.us About Blog Contact Class01 Class02 Class03 Class04 Class05 Class06 Class07 Class08 Class09 Class10 dan101 Forum Google Hangout Vboxen