## Aggregate Data

This lab required some knowledge.

I knew the price data I get from truefx.com is too fine-grained.

Many of the observations are spaced microseconds apart.

I wanted to have observations spaced 5 minutes apart.

My intent was to learn price behavior for each pair for a variety of holding periods.

From a trading perspective, the holding period which interested me the most is one hour.

The main idea I implement here is that for each pair I should create groups of observations which are spaced 5 minutes apart.

Then I should use groupby() in Pandas to find mean of price for each group of observations.

I started this lab with a simple shell command:

``mkdir -p ~fx411/csv/``

Next I wrote the script listed below:

``````
# agg_zip.py

# This script should read zip files.
# For each file it should compute current price from (bid+ask)/2.
# Next, it should groupby timestamps truncated to 5 minute boundries.
# Finally it should output a CSV file with mean of current_price for each 5 minute boundry.

import pandas as pd
import glob
fn_l = sorted(glob.glob("/home/fx411/truefx/*USD*20*.zip"))
for fn_s in fn_l:
print(fn_s[19:33])
pairmo_s     = fn_s[19:33]
zipf         = '/home/fx411/truefx/'+pairmo_s+'.zip'
f11_df       =  f10_df.copy()[['pair','ts0']]
ts1_l        = [ts[:14] for ts in f11_df.ts0]
ts2_sr       = pd.to_datetime(ts1_l, utc=True)
f11_df['ts'] = [5*60*int(int(ts.strftime("%s"))/5/60) for ts in ts2_sr]
f12_df       = f11_df.copy()[['ts','cp']]
f13_df = f12_df.groupby(['ts']).cp.mean()
f13_df.to_csv('/home/fx411/csv/'+pairmo_s+'.csv', float_format='%4.6f')

'bye'
``````

A copy of the above script should be in the git repo at this location:

~fx411/fx411/script/agg_zip.py

Also I should be able to see it here:

I ran the above script.

It needed about 5.5 hours to process 9.3 GB of compressed price data.

You can run the script on your laptop to access the aggregated data.

Or you can download the aggregated data from this site:

The resulting aggregated data consumed only 54 MB of space.

A list of the aggregated data is displayed below:

``````
fx411@ub16aug:~\$
fx411@ub16aug:~\$ cd ~fx411
fx411@ub16aug:~\$
fx411@ub16aug:~\$ du -sh csv/
54M	csv/
fx411@ub16aug:~\$
fx411@ub16aug:~\$
fx411@ub16aug:~\$ ls -la csv/
total 54340
drwxrwxr-x  2 fx411 fx411  20480 Nov  8 09:27 .
drwxr-xr-x 28 fx411 fx411   4096 Nov  8 14:10 ..
-rw-rw-r--  1 fx411 fx411 115060 Nov  8 03:49 AUDUSD-2009-05.csv
-rw-rw-r--  1 fx411 fx411 109120 Nov  8 03:50 AUDUSD-2009-06.csv
-rw-rw-r--  1 fx411 fx411 131580 Nov  8 03:50 AUDUSD-2009-07.csv
-rw-rw-r--  1 fx411 fx411 120860 Nov  8 03:51 AUDUSD-2009-08.csv
-rw-rw-r--  1 fx411 fx411 115000 Nov  8 03:51 AUDUSD-2009-09.csv
-rw-rw-r--  1 fx411 fx411 125880 Nov  8 03:52 AUDUSD-2009-10.csv
-rw-rw-r--  1 fx411 fx411 115520 Nov  8 03:52 AUDUSD-2009-11.csv
-rw-rw-r--  1 fx411 fx411 128440 Nov  8 03:52 AUDUSD-2009-12.csv
-rw-rw-r--  1 fx411 fx411 116200 Nov  8 03:52 AUDUSD-2010-01.csv
-rw-rw-r--  1 fx411 fx411  81000 Nov  8 03:53 AUDUSD-2010-02.csv
-rw-rw-r--  1 fx411 fx411 126920 Nov  8 03:53 AUDUSD-2010-03.csv
-rw-rw-r--  1 fx411 fx411 125500 Nov  8 03:53 AUDUSD-2010-04.csv
-rw-rw-r--  1 fx411 fx411 121560 Nov  8 03:54 AUDUSD-2010-05.csv
-rw-rw-r--  1 fx411 fx411 117440 Nov  8 03:55 AUDUSD-2010-06.csv
-rw-rw-r--  1 fx411 fx411 125840 Nov  8 03:55 AUDUSD-2010-07.csv
-rw-rw-r--  1 fx411 fx411 127220 Nov  8 03:56 AUDUSD-2010-08.csv
-rw-rw-r--  1 fx411 fx411 126580 Nov  8 03:56 AUDUSD-2010-09.csv
-rw-rw-r--  1 fx411 fx411 120480 Nov  8 03:56 AUDUSD-2010-10.csv
-rw-rw-r--  1 fx411 fx411 126880 Nov  8 03:57 AUDUSD-2010-11.csv
-rw-rw-r--  1 fx411 fx411 132420 Nov  8 03:57 AUDUSD-2010-12.csv
-rw-rw-r--  1 fx411 fx411 106980 Nov  8 03:57 AUDUSD-2011-01.csv
-rw-rw-r--  1 fx411 fx411 115780 Nov  8 03:58 AUDUSD-2011-02.csv
-rw-rw-r--  1 fx411 fx411 133400 Nov  8 03:58 AUDUSD-2011-03.csv
-rw-rw-r--  1 fx411 fx411 120160 Nov  8 03:58 AUDUSD-2011-04.csv
-rw-rw-r--  1 fx411 fx411 115540 Nov  8 03:59 AUDUSD-2011-05.csv
-rw-rw-r--  1 fx411 fx411 120820 Nov  8 04:00 AUDUSD-2011-06.csv
-rw-rw-r--  1 fx411 fx411 120000 Nov  8 04:00 AUDUSD-2011-07.csv
-rw-rw-r--  1 fx411 fx411 114980 Nov  8 04:01 AUDUSD-2011-08.csv
-rw-rw-r--  1 fx411 fx411 125900 Nov  8 04:02 AUDUSD-2011-09.csv
-rw-rw-r--  1 fx411 fx411 121200 Nov  8 04:03 AUDUSD-2011-10.csv
-rw-rw-r--  1 fx411 fx411 127540 Nov  8 04:03 AUDUSD-2011-11.csv
-rw-rw-r--  1 fx411 fx411 126680 Nov  8 04:04 AUDUSD-2011-12.csv
-rw-rw-r--  1 fx411 fx411 118460 Nov  8 04:05 AUDUSD-2012-01.csv
-rw-rw-r--  1 fx411 fx411 117400 Nov  8 04:05 AUDUSD-2012-02.csv
-rw-rw-r--  1 fx411 fx411 126300 Nov  8 04:06 AUDUSD-2012-03.csv
-rw-rw-r--  1 fx411 fx411 121340 Nov  8 04:07 AUDUSD-2012-04.csv
-rw-rw-r--  1 fx411 fx411 127740 Nov  8 04:08 AUDUSD-2012-05.csv
-rw-rw-r--  1 fx411 fx411 110880 Nov  8 04:10 AUDUSD-2012-06.csv
-rw-rw-r--  1 fx411 fx411 127240 Nov  8 04:12 AUDUSD-2012-07.csv
-rw-rw-r--  1 fx411 fx411 127820 Nov  8 04:14 AUDUSD-2012-08.csv
-rw-rw-r--  1 fx411 fx411 112320 Nov  8 04:15 AUDUSD-2012-09.csv
-rw-rw-r--  1 fx411 fx411 129440 Nov  8 04:16 AUDUSD-2012-10.csv
-rw-rw-r--  1 fx411 fx411 124040 Nov  8 04:17 AUDUSD-2012-11.csv
-rw-rw-r--  1 fx411 fx411 115300 Nov  8 04:17 AUDUSD-2012-12.csv
-rw-rw-r--  1 fx411 fx411 125360 Nov  8 04:18 AUDUSD-2013-01.csv
-rw-rw-r--  1 fx411 fx411 113300 Nov  8 04:18 AUDUSD-2013-02.csv
-rw-rw-r--  1 fx411 fx411 116440 Nov  8 04:19 AUDUSD-2013-03.csv
-rw-rw-r--  1 fx411 fx411 123840 Nov  8 04:20 AUDUSD-2013-04.csv
-rw-rw-r--  1 fx411 fx411 128880 Nov  8 04:21 AUDUSD-2013-05.csv
-rw-rw-r--  1 fx411 fx411 105600 Nov  8 04:22 AUDUSD-2013-06.csv
-rw-rw-r--  1 fx411 fx411 128100 Nov  8 04:23 AUDUSD-2013-07.csv
-rw-rw-r--  1 fx411 fx411 121460 Nov  8 04:24 AUDUSD-2013-08.csv
-rw-rw-r--  1 fx411 fx411 116840 Nov  8 04:24 AUDUSD-2013-09.csv
-rw-rw-r--  1 fx411 fx411 129540 Nov  8 04:25 AUDUSD-2013-10.csv
-rw-rw-r--  1 fx411 fx411 112600 Nov  8 04:25 AUDUSD-2013-11.csv
-rw-rw-r--  1 fx411 fx411 120960 Nov  8 04:25 AUDUSD-2013-12.csv
-rw-rw-r--  1 fx411 fx411 124840 Nov  8 04:25 AUDUSD-2014-01.csv
-rw-rw-r--  1 fx411 fx411 113300 Nov  8 04:26 AUDUSD-2014-02.csv
-rw-rw-r--  1 fx411 fx411 117920 Nov  8 04:26 AUDUSD-2014-03.csv
-rw-rw-r--  1 fx411 fx411 123880 Nov  8 04:26 AUDUSD-2014-04.csv
-rw-rw-r--  1 fx411 fx411 123180 Nov  8 04:27 AUDUSD-2014-05.csv
-rw-rw-r--  1 fx411 fx411 118100 Nov  8 04:27 AUDUSD-2014-06.csv
-rw-rw-r--  1 fx411 fx411 129620 Nov  8 04:27 AUDUSD-2014-07.csv
-rw-rw-r--  1 fx411 fx411 117420 Nov  8 04:27 AUDUSD-2014-08.csv
-rw-rw-r--  1 fx411 fx411 113820 Nov  8 04:28 AUDUSD-2014-09.csv
-rw-rw-r--  1 fx411 fx411 123200 Nov  8 04:28 AUDUSD-2014-10.csv
-rw-rw-r--  1 fx411 fx411  90400 Nov  8 04:29 AUDUSD-2014-11.csv
-rw-rw-r--  1 fx411 fx411  31860 Nov  8 04:29 AUDUSD-2014-12.csv
-rw-rw-r--  1 fx411 fx411 119200 Nov  8 04:29 AUDUSD-2015-01.csv
-rw-rw-r--  1 fx411 fx411 113260 Nov  8 04:30 AUDUSD-2015-02.csv
-rw-rw-r--  1 fx411 fx411 124120 Nov  8 04:30 AUDUSD-2015-03.csv
-rw-rw-r--  1 fx411 fx411 118440 Nov  8 04:31 AUDUSD-2015-04.csv
-rw-rw-r--  1 fx411 fx411 118200 Nov  8 04:31 AUDUSD-2015-05.csv
-rw-rw-r--  1 fx411 fx411 123860 Nov  8 04:32 AUDUSD-2015-06.csv
-rw-rw-r--  1 fx411 fx411 128880 Nov  8 04:32 AUDUSD-2015-07.csv
-rw-rw-r--  1 fx411 fx411 118100 Nov  8 04:33 AUDUSD-2015-08.csv
-rw-rw-r--  1 fx411 fx411 123860 Nov  8 04:34 AUDUSD-2015-09.csv
-rw-rw-r--  1 fx411 fx411 123140 Nov  8 04:34 AUDUSD-2015-10.csv
-rw-rw-r--  1 fx411 fx411 119040 Nov  8 04:34 AUDUSD-2015-11.csv
-rw-rw-r--  1 fx411 fx411 126640 Nov  8 04:35 AUDUSD-2015-12.csv
-rw-rw-r--  1 fx411 fx411 113040 Nov  8 04:35 AUDUSD-2016-01.csv
-rw-rw-r--  1 fx411 fx411 118560 Nov  8 04:36 AUDUSD-2016-02.csv
-rw-rw-r--  1 fx411 fx411 129600 Nov  8 04:36 AUDUSD-2016-03.csv
-rw-rw-r--  1 fx411 fx411 117400 Nov  8 04:38 AUDUSD-2016-04.csv
-rw-rw-r--  1 fx411 fx411 123840 Nov  8 04:41 AUDUSD-2016-05.csv
-rw-rw-r--  1 fx411 fx411 123800 Nov  8 04:44 AUDUSD-2016-06.csv
-rw-rw-r--  1 fx411 fx411 117360 Nov  8 04:47 AUDUSD-2016-07.csv
-rw-rw-r--  1 fx411 fx411 129600 Nov  8 04:49 AUDUSD-2016-08.csv
-rw-rw-r--  1 fx411 fx411 123120 Nov  8 04:51 AUDUSD-2016-09.csv
-rw-rw-r--  1 fx411 fx411 118040 Nov  8 04:53 AUDUSD-2016-10.csv
-rw-rw-r--  1 fx411 fx411 115020 Nov  8 04:54 EURUSD-2009-05.csv
-rw-rw-r--  1 fx411 fx411 109240 Nov  8 04:55 EURUSD-2009-06.csv
-rw-rw-r--  1 fx411 fx411 131540 Nov  8 04:55 EURUSD-2009-07.csv
-rw-rw-r--  1 fx411 fx411 120760 Nov  8 04:56 EURUSD-2009-08.csv
-rw-rw-r--  1 fx411 fx411 115020 Nov  8 04:56 EURUSD-2009-09.csv
-rw-rw-r--  1 fx411 fx411 125860 Nov  8 04:57 EURUSD-2009-10.csv
-rw-rw-r--  1 fx411 fx411 115520 Nov  8 04:57 EURUSD-2009-11.csv
-rw-rw-r--  1 fx411 fx411 128760 Nov  8 04:58 EURUSD-2009-12.csv
-rw-rw-r--  1 fx411 fx411 116340 Nov  8 04:58 EURUSD-2010-01.csv
-rw-rw-r--  1 fx411 fx411  81000 Nov  8 04:59 EURUSD-2010-02.csv
-rw-rw-r--  1 fx411 fx411 126940 Nov  8 04:59 EURUSD-2010-03.csv
-rw-rw-r--  1 fx411 fx411 125560 Nov  8 05:00 EURUSD-2010-04.csv
-rw-rw-r--  1 fx411 fx411 121580 Nov  8 05:01 EURUSD-2010-05.csv
-rw-rw-r--  1 fx411 fx411 117440 Nov  8 05:02 EURUSD-2010-06.csv
-rw-rw-r--  1 fx411 fx411 125860 Nov  8 05:02 EURUSD-2010-07.csv
-rw-rw-r--  1 fx411 fx411 127260 Nov  8 05:03 EURUSD-2010-08.csv
-rw-rw-r--  1 fx411 fx411 126640 Nov  8 05:03 EURUSD-2010-09.csv
-rw-rw-r--  1 fx411 fx411 120840 Nov  8 05:04 EURUSD-2010-10.csv
-rw-rw-r--  1 fx411 fx411 126880 Nov  8 05:05 EURUSD-2010-11.csv
-rw-rw-r--  1 fx411 fx411 132420 Nov  8 05:05 EURUSD-2010-12.csv
-rw-rw-r--  1 fx411 fx411 112740 Nov  8 05:06 EURUSD-2011-01.csv
-rw-rw-r--  1 fx411 fx411 115820 Nov  8 05:07 EURUSD-2011-02.csv
-rw-rw-r--  1 fx411 fx411 133520 Nov  8 05:08 EURUSD-2011-03.csv
-rw-rw-r--  1 fx411 fx411 120160 Nov  8 05:09 EURUSD-2011-04.csv
-rw-rw-r--  1 fx411 fx411 115540 Nov  8 05:10 EURUSD-2011-05.csv
-rw-rw-r--  1 fx411 fx411 120800 Nov  8 05:12 EURUSD-2011-06.csv
-rw-rw-r--  1 fx411 fx411 120020 Nov  8 05:14 EURUSD-2011-07.csv
-rw-rw-r--  1 fx411 fx411 114960 Nov  8 05:16 EURUSD-2011-08.csv
-rw-rw-r--  1 fx411 fx411 125880 Nov  8 05:18 EURUSD-2011-09.csv
-rw-rw-r--  1 fx411 fx411 121260 Nov  8 05:20 EURUSD-2011-10.csv
-rw-rw-r--  1 fx411 fx411 127540 Nov  8 05:22 EURUSD-2011-11.csv
-rw-rw-r--  1 fx411 fx411 127180 Nov  8 05:23 EURUSD-2011-12.csv
-rw-rw-r--  1 fx411 fx411 118500 Nov  8 05:24 EURUSD-2012-01.csv
-rw-rw-r--  1 fx411 fx411 117400 Nov  8 05:26 EURUSD-2012-02.csv
-rw-rw-r--  1 fx411 fx411 126260 Nov  8 05:27 EURUSD-2012-03.csv
-rw-rw-r--  1 fx411 fx411 121400 Nov  8 05:29 EURUSD-2012-04.csv
-rw-rw-r--  1 fx411 fx411 127740 Nov  8 05:31 EURUSD-2012-05.csv
-rw-rw-r--  1 fx411 fx411 110880 Nov  8 05:34 EURUSD-2012-06.csv
-rw-rw-r--  1 fx411 fx411 127240 Nov  8 05:36 EURUSD-2012-07.csv
-rw-rw-r--  1 fx411 fx411 128060 Nov  8 05:39 EURUSD-2012-08.csv
-rw-rw-r--  1 fx411 fx411 112320 Nov  8 05:41 EURUSD-2012-09.csv
-rw-rw-r--  1 fx411 fx411 129440 Nov  8 05:42 EURUSD-2012-10.csv
-rw-rw-r--  1 fx411 fx411 124080 Nov  8 05:44 EURUSD-2012-11.csv
-rw-rw-r--  1 fx411 fx411 115340 Nov  8 05:45 EURUSD-2012-12.csv
-rw-rw-r--  1 fx411 fx411 119620 Nov  8 05:47 EURUSD-2013-01.csv
-rw-rw-r--  1 fx411 fx411 113360 Nov  8 05:50 EURUSD-2013-02.csv
-rw-rw-r--  1 fx411 fx411 116460 Nov  8 05:52 EURUSD-2013-03.csv
-rw-rw-r--  1 fx411 fx411 123880 Nov  8 05:55 EURUSD-2013-04.csv
-rw-rw-r--  1 fx411 fx411 128880 Nov  8 05:56 EURUSD-2013-05.csv
-rw-rw-r--  1 fx411 fx411 105600 Nov  8 05:57 EURUSD-2013-06.csv
-rw-rw-r--  1 fx411 fx411 128120 Nov  8 05:59 EURUSD-2013-07.csv
-rw-rw-r--  1 fx411 fx411 121440 Nov  8 06:00 EURUSD-2013-08.csv
-rw-rw-r--  1 fx411 fx411 116840 Nov  8 06:00 EURUSD-2013-09.csv
-rw-rw-r--  1 fx411 fx411 129580 Nov  8 06:01 EURUSD-2013-10.csv
-rw-rw-r--  1 fx411 fx411 112600 Nov  8 06:01 EURUSD-2013-11.csv
-rw-rw-r--  1 fx411 fx411 120960 Nov  8 06:01 EURUSD-2013-12.csv
-rw-rw-r--  1 fx411 fx411 124820 Nov  8 06:02 EURUSD-2014-01.csv
-rw-rw-r--  1 fx411 fx411 113340 Nov  8 06:02 EURUSD-2014-02.csv
-rw-rw-r--  1 fx411 fx411 117920 Nov  8 06:03 EURUSD-2014-03.csv
-rw-rw-r--  1 fx411 fx411 123920 Nov  8 06:03 EURUSD-2014-04.csv
-rw-rw-r--  1 fx411 fx411 123180 Nov  8 06:03 EURUSD-2014-05.csv
-rw-rw-r--  1 fx411 fx411 118140 Nov  8 06:04 EURUSD-2014-06.csv
-rw-rw-r--  1 fx411 fx411 129600 Nov  8 06:04 EURUSD-2014-07.csv
-rw-rw-r--  1 fx411 fx411 117420 Nov  8 06:04 EURUSD-2014-08.csv
-rw-rw-r--  1 fx411 fx411 119100 Nov  8 06:05 EURUSD-2014-09.csv
-rw-rw-r--  1 fx411 fx411 123200 Nov  8 06:06 EURUSD-2014-10.csv
-rw-rw-r--  1 fx411 fx411  90380 Nov  8 06:06 EURUSD-2014-11.csv
-rw-rw-r--  1 fx411 fx411  60380 Nov  8 06:06 EURUSD-2014-12.csv
-rw-rw-r--  1 fx411 fx411 119220 Nov  8 06:07 EURUSD-2015-01.csv
-rw-rw-r--  1 fx411 fx411 113320 Nov  8 06:08 EURUSD-2015-02.csv
-rw-rw-r--  1 fx411 fx411 124160 Nov  8 06:08 EURUSD-2015-03.csv
-rw-rw-r--  1 fx411 fx411 118460 Nov  8 06:09 EURUSD-2015-04.csv
-rw-rw-r--  1 fx411 fx411 118240 Nov  8 06:10 EURUSD-2015-05.csv
-rw-rw-r--  1 fx411 fx411 123840 Nov  8 06:11 EURUSD-2015-06.csv
-rw-rw-r--  1 fx411 fx411 128900 Nov  8 06:12 EURUSD-2015-07.csv
-rw-rw-r--  1 fx411 fx411 118100 Nov  8 06:13 EURUSD-2015-08.csv
-rw-rw-r--  1 fx411 fx411 123860 Nov  8 06:14 EURUSD-2015-09.csv
-rw-rw-r--  1 fx411 fx411 123120 Nov  8 06:14 EURUSD-2015-10.csv
-rw-rw-r--  1 fx411 fx411 119040 Nov  8 06:15 EURUSD-2015-11.csv
-rw-rw-r--  1 fx411 fx411 126700 Nov  8 06:15 EURUSD-2015-12.csv
-rw-rw-r--  1 fx411 fx411 113040 Nov  8 06:16 EURUSD-2016-01.csv
-rw-rw-r--  1 fx411 fx411 118560 Nov  8 06:17 EURUSD-2016-02.csv
-rw-rw-r--  1 fx411 fx411 129600 Nov  8 06:18 EURUSD-2016-03.csv
-rw-rw-r--  1 fx411 fx411 117420 Nov  8 06:19 EURUSD-2016-04.csv
-rw-rw-r--  1 fx411 fx411 123840 Nov  8 06:21 EURUSD-2016-05.csv
-rw-rw-r--  1 fx411 fx411 123800 Nov  8 06:25 EURUSD-2016-06.csv
-rw-rw-r--  1 fx411 fx411 117360 Nov  8 06:27 EURUSD-2016-07.csv
-rw-rw-r--  1 fx411 fx411 129600 Nov  8 06:29 EURUSD-2016-08.csv
-rw-rw-r--  1 fx411 fx411 123120 Nov  8 06:31 EURUSD-2016-09.csv
-rw-rw-r--  1 fx411 fx411 118020 Nov  8 06:33 EURUSD-2016-10.csv
-rw-rw-r--  1 fx411 fx411 115000 Nov  8 06:34 GBPUSD-2009-05.csv
-rw-rw-r--  1 fx411 fx411 109120 Nov  8 06:34 GBPUSD-2009-06.csv
-rw-rw-r--  1 fx411 fx411 131580 Nov  8 06:35 GBPUSD-2009-07.csv
-rw-rw-r--  1 fx411 fx411 120800 Nov  8 06:35 GBPUSD-2009-08.csv
-rw-rw-r--  1 fx411 fx411 115000 Nov  8 06:36 GBPUSD-2009-09.csv
-rw-rw-r--  1 fx411 fx411 125820 Nov  8 06:36 GBPUSD-2009-10.csv
-rw-rw-r--  1 fx411 fx411 115480 Nov  8 06:37 GBPUSD-2009-11.csv
-rw-rw-r--  1 fx411 fx411 128500 Nov  8 06:37 GBPUSD-2009-12.csv
-rw-rw-r--  1 fx411 fx411 116300 Nov  8 06:37 GBPUSD-2010-01.csv
-rw-rw-r--  1 fx411 fx411  81000 Nov  8 06:38 GBPUSD-2010-02.csv
-rw-rw-r--  1 fx411 fx411 126820 Nov  8 06:38 GBPUSD-2010-03.csv
-rw-rw-r--  1 fx411 fx411 125400 Nov  8 06:39 GBPUSD-2010-04.csv
-rw-rw-r--  1 fx411 fx411 121480 Nov  8 06:39 GBPUSD-2010-05.csv
-rw-rw-r--  1 fx411 fx411 117400 Nov  8 06:40 GBPUSD-2010-06.csv
-rw-rw-r--  1 fx411 fx411 125840 Nov  8 06:41 GBPUSD-2010-07.csv
-rw-rw-r--  1 fx411 fx411 127220 Nov  8 06:41 GBPUSD-2010-08.csv
-rw-rw-r--  1 fx411 fx411 126480 Nov  8 06:41 GBPUSD-2010-09.csv
-rw-rw-r--  1 fx411 fx411 120820 Nov  8 06:42 GBPUSD-2010-10.csv
-rw-rw-r--  1 fx411 fx411 126920 Nov  8 06:42 GBPUSD-2010-11.csv
-rw-rw-r--  1 fx411 fx411 132420 Nov  8 06:42 GBPUSD-2010-12.csv
-rw-rw-r--  1 fx411 fx411 112720 Nov  8 06:43 GBPUSD-2011-01.csv
-rw-rw-r--  1 fx411 fx411 115820 Nov  8 06:43 GBPUSD-2011-02.csv
-rw-rw-r--  1 fx411 fx411 133460 Nov  8 06:44 GBPUSD-2011-03.csv
-rw-rw-r--  1 fx411 fx411 120040 Nov  8 06:44 GBPUSD-2011-04.csv
-rw-rw-r--  1 fx411 fx411 115480 Nov  8 06:45 GBPUSD-2011-05.csv
-rw-rw-r--  1 fx411 fx411 120700 Nov  8 06:45 GBPUSD-2011-06.csv
-rw-rw-r--  1 fx411 fx411 119960 Nov  8 06:46 GBPUSD-2011-07.csv
-rw-rw-r--  1 fx411 fx411 114960 Nov  8 06:47 GBPUSD-2011-08.csv
-rw-rw-r--  1 fx411 fx411 125800 Nov  8 06:47 GBPUSD-2011-09.csv
-rw-rw-r--  1 fx411 fx411 120920 Nov  8 06:48 GBPUSD-2011-10.csv
-rw-rw-r--  1 fx411 fx411 127360 Nov  8 06:48 GBPUSD-2011-11.csv
-rw-rw-r--  1 fx411 fx411 126880 Nov  8 06:49 GBPUSD-2011-12.csv
-rw-rw-r--  1 fx411 fx411 118340 Nov  8 06:49 GBPUSD-2012-01.csv
-rw-rw-r--  1 fx411 fx411 117300 Nov  8 06:49 GBPUSD-2012-02.csv
-rw-rw-r--  1 fx411 fx411 126140 Nov  8 06:50 GBPUSD-2012-03.csv
-rw-rw-r--  1 fx411 fx411 121220 Nov  8 06:50 GBPUSD-2012-04.csv
-rw-rw-r--  1 fx411 fx411 127640 Nov  8 06:51 GBPUSD-2012-05.csv
-rw-rw-r--  1 fx411 fx411 110740 Nov  8 06:53 GBPUSD-2012-06.csv
-rw-rw-r--  1 fx411 fx411 127140 Nov  8 06:54 GBPUSD-2012-07.csv
-rw-rw-r--  1 fx411 fx411 127880 Nov  8 06:55 GBPUSD-2012-08.csv
-rw-rw-r--  1 fx411 fx411 112320 Nov  8 06:56 GBPUSD-2012-09.csv
-rw-rw-r--  1 fx411 fx411 129440 Nov  8 06:57 GBPUSD-2012-10.csv
-rw-rw-r--  1 fx411 fx411 124080 Nov  8 06:58 GBPUSD-2012-11.csv
-rw-rw-r--  1 fx411 fx411 115340 Nov  8 06:58 GBPUSD-2012-12.csv
-rw-rw-r--  1 fx411 fx411 125380 Nov  8 06:59 GBPUSD-2013-01.csv
-rw-rw-r--  1 fx411 fx411 113360 Nov  8 07:00 GBPUSD-2013-02.csv
-rw-rw-r--  1 fx411 fx411 116440 Nov  8 07:01 GBPUSD-2013-03.csv
-rw-rw-r--  1 fx411 fx411 118120 Nov  8 07:02 GBPUSD-2013-04.csv
-rw-rw-r--  1 fx411 fx411 128840 Nov  8 07:03 GBPUSD-2013-05.csv
-rw-rw-r--  1 fx411 fx411 105600 Nov  8 07:04 GBPUSD-2013-06.csv
-rw-rw-r--  1 fx411 fx411 128140 Nov  8 07:05 GBPUSD-2013-07.csv
-rw-rw-r--  1 fx411 fx411 121440 Nov  8 07:06 GBPUSD-2013-08.csv
-rw-rw-r--  1 fx411 fx411 116840 Nov  8 07:07 GBPUSD-2013-09.csv
-rw-rw-r--  1 fx411 fx411 129580 Nov  8 07:07 GBPUSD-2013-10.csv
-rw-rw-r--  1 fx411 fx411 118320 Nov  8 07:07 GBPUSD-2013-11.csv
-rw-rw-r--  1 fx411 fx411 120960 Nov  8 07:07 GBPUSD-2013-12.csv
-rw-rw-r--  1 fx411 fx411 124840 Nov  8 07:08 GBPUSD-2014-01.csv
-rw-rw-r--  1 fx411 fx411 113320 Nov  8 07:08 GBPUSD-2014-02.csv
-rw-rw-r--  1 fx411 fx411 117900 Nov  8 07:08 GBPUSD-2014-03.csv
-rw-rw-r--  1 fx411 fx411 123860 Nov  8 07:09 GBPUSD-2014-04.csv
-rw-rw-r--  1 fx411 fx411 123180 Nov  8 07:09 GBPUSD-2014-05.csv
-rw-rw-r--  1 fx411 fx411 118120 Nov  8 07:09 GBPUSD-2014-06.csv
-rw-rw-r--  1 fx411 fx411 129620 Nov  8 07:09 GBPUSD-2014-07.csv
-rw-rw-r--  1 fx411 fx411 117460 Nov  8 07:10 GBPUSD-2014-08.csv
-rw-rw-r--  1 fx411 fx411 119100 Nov  8 07:10 GBPUSD-2014-09.csv
-rw-rw-r--  1 fx411 fx411 112440 Nov  8 07:10 GBPUSD-2014-10.csv
-rw-rw-r--  1 fx411 fx411  74100 Nov  8 07:11 GBPUSD-2014-11.csv
-rw-rw-r--  1 fx411 fx411  87400 Nov  8 07:11 GBPUSD-2014-12.csv
-rw-rw-r--  1 fx411 fx411 119180 Nov  8 07:11 GBPUSD-2015-01.csv
-rw-rw-r--  1 fx411 fx411 113300 Nov  8 07:12 GBPUSD-2015-02.csv
-rw-rw-r--  1 fx411 fx411 124060 Nov  8 07:12 GBPUSD-2015-03.csv
-rw-rw-r--  1 fx411 fx411 118420 Nov  8 07:13 GBPUSD-2015-04.csv
-rw-rw-r--  1 fx411 fx411 118200 Nov  8 07:13 GBPUSD-2015-05.csv
-rw-rw-r--  1 fx411 fx411 123840 Nov  8 07:14 GBPUSD-2015-06.csv
-rw-rw-r--  1 fx411 fx411 128880 Nov  8 07:15 GBPUSD-2015-07.csv
-rw-rw-r--  1 fx411 fx411 118100 Nov  8 07:16 GBPUSD-2015-08.csv
-rw-rw-r--  1 fx411 fx411 123820 Nov  8 07:16 GBPUSD-2015-09.csv
-rw-rw-r--  1 fx411 fx411 123120 Nov  8 07:17 GBPUSD-2015-10.csv
-rw-rw-r--  1 fx411 fx411 119020 Nov  8 07:17 GBPUSD-2015-11.csv
-rw-rw-r--  1 fx411 fx411 126620 Nov  8 07:18 GBPUSD-2015-12.csv
-rw-rw-r--  1 fx411 fx411 112800 Nov  8 07:18 GBPUSD-2016-01.csv
-rw-rw-r--  1 fx411 fx411 118320 Nov  8 07:19 GBPUSD-2016-02.csv
-rw-rw-r--  1 fx411 fx411 129600 Nov  8 07:20 GBPUSD-2016-03.csv
-rw-rw-r--  1 fx411 fx411 117380 Nov  8 07:21 GBPUSD-2016-04.csv
-rw-rw-r--  1 fx411 fx411 123840 Nov  8 07:24 GBPUSD-2016-05.csv
-rw-rw-r--  1 fx411 fx411 123800 Nov  8 07:27 GBPUSD-2016-06.csv
-rw-rw-r--  1 fx411 fx411 117360 Nov  8 07:30 GBPUSD-2016-07.csv
-rw-rw-r--  1 fx411 fx411 129600 Nov  8 07:32 GBPUSD-2016-08.csv
-rw-rw-r--  1 fx411 fx411 123120 Nov  8 07:34 GBPUSD-2016-09.csv
-rw-rw-r--  1 fx411 fx411 118040 Nov  8 07:36 GBPUSD-2016-10.csv
-rw-rw-r--  1 fx411 fx411 114820 Nov  8 07:37 USDCAD-2009-05.csv
-rw-rw-r--  1 fx411 fx411 108920 Nov  8 07:37 USDCAD-2009-06.csv
-rw-rw-r--  1 fx411 fx411 131240 Nov  8 07:37 USDCAD-2009-07.csv
-rw-rw-r--  1 fx411 fx411 120600 Nov  8 07:38 USDCAD-2009-08.csv
-rw-rw-r--  1 fx411 fx411 114740 Nov  8 07:38 USDCAD-2009-09.csv
-rw-rw-r--  1 fx411 fx411 125580 Nov  8 07:38 USDCAD-2009-10.csv
-rw-rw-r--  1 fx411 fx411 115360 Nov  8 07:39 USDCAD-2009-11.csv
-rw-rw-r--  1 fx411 fx411 127760 Nov  8 07:39 USDCAD-2009-12.csv
-rw-rw-r--  1 fx411 fx411 116140 Nov  8 07:39 USDCAD-2010-01.csv
-rw-rw-r--  1 fx411 fx411  80840 Nov  8 07:39 USDCAD-2010-02.csv
-rw-rw-r--  1 fx411 fx411 126700 Nov  8 07:39 USDCAD-2010-03.csv
-rw-rw-r--  1 fx411 fx411 125300 Nov  8 07:40 USDCAD-2010-04.csv
-rw-rw-r--  1 fx411 fx411 121400 Nov  8 07:40 USDCAD-2010-05.csv
-rw-rw-r--  1 fx411 fx411 117300 Nov  8 07:41 USDCAD-2010-06.csv
-rw-rw-r--  1 fx411 fx411 125780 Nov  8 07:41 USDCAD-2010-07.csv
-rw-rw-r--  1 fx411 fx411 127160 Nov  8 07:42 USDCAD-2010-08.csv
-rw-rw-r--  1 fx411 fx411 126400 Nov  8 07:42 USDCAD-2010-09.csv
-rw-rw-r--  1 fx411 fx411 120220 Nov  8 07:42 USDCAD-2010-10.csv
-rw-rw-r--  1 fx411 fx411 126740 Nov  8 07:43 USDCAD-2010-11.csv
-rw-rw-r--  1 fx411 fx411 132260 Nov  8 07:43 USDCAD-2010-12.csv
-rw-rw-r--  1 fx411 fx411 106920 Nov  8 07:43 USDCAD-2011-01.csv
-rw-rw-r--  1 fx411 fx411 115740 Nov  8 07:43 USDCAD-2011-02.csv
-rw-rw-r--  1 fx411 fx411 133340 Nov  8 07:44 USDCAD-2011-03.csv
-rw-rw-r--  1 fx411 fx411 119980 Nov  8 07:44 USDCAD-2011-04.csv
-rw-rw-r--  1 fx411 fx411 115460 Nov  8 07:44 USDCAD-2011-05.csv
-rw-rw-r--  1 fx411 fx411 120720 Nov  8 07:44 USDCAD-2011-06.csv
-rw-rw-r--  1 fx411 fx411 119920 Nov  8 07:45 USDCAD-2011-07.csv
-rw-rw-r--  1 fx411 fx411 114920 Nov  8 07:45 USDCAD-2011-08.csv
-rw-rw-r--  1 fx411 fx411 125880 Nov  8 07:46 USDCAD-2011-09.csv
-rw-rw-r--  1 fx411 fx411 120780 Nov  8 07:46 USDCAD-2011-10.csv
-rw-rw-r--  1 fx411 fx411 127380 Nov  8 07:47 USDCAD-2011-11.csv
-rw-rw-r--  1 fx411 fx411 126580 Nov  8 07:47 USDCAD-2011-12.csv
-rw-rw-r--  1 fx411 fx411 117860 Nov  8 07:47 USDCAD-2012-01.csv
-rw-rw-r--  1 fx411 fx411 117200 Nov  8 07:48 USDCAD-2012-02.csv
-rw-rw-r--  1 fx411 fx411 126000 Nov  8 07:48 USDCAD-2012-03.csv
-rw-rw-r--  1 fx411 fx411 121140 Nov  8 07:48 USDCAD-2012-04.csv
-rw-rw-r--  1 fx411 fx411 127600 Nov  8 07:48 USDCAD-2012-05.csv
-rw-rw-r--  1 fx411 fx411 110820 Nov  8 07:49 USDCAD-2012-06.csv
-rw-rw-r--  1 fx411 fx411 126960 Nov  8 07:50 USDCAD-2012-07.csv
-rw-rw-r--  1 fx411 fx411 127800 Nov  8 07:50 USDCAD-2012-08.csv
-rw-rw-r--  1 fx411 fx411 112260 Nov  8 07:51 USDCAD-2012-09.csv
-rw-rw-r--  1 fx411 fx411 129440 Nov  8 07:51 USDCAD-2012-10.csv
-rw-rw-r--  1 fx411 fx411 124080 Nov  8 07:52 USDCAD-2012-11.csv
-rw-rw-r--  1 fx411 fx411 115120 Nov  8 07:52 USDCAD-2012-12.csv
-rw-rw-r--  1 fx411 fx411 125300 Nov  8 07:52 USDCAD-2013-01.csv
-rw-rw-r--  1 fx411 fx411 113320 Nov  8 07:53 USDCAD-2013-02.csv
-rw-rw-r--  1 fx411 fx411 116420 Nov  8 07:53 USDCAD-2013-03.csv
-rw-rw-r--  1 fx411 fx411 123840 Nov  8 07:53 USDCAD-2013-04.csv
-rw-rw-r--  1 fx411 fx411 128880 Nov  8 07:54 USDCAD-2013-05.csv
-rw-rw-r--  1 fx411 fx411 105600 Nov  8 07:54 USDCAD-2013-06.csv
-rw-rw-r--  1 fx411 fx411 128120 Nov  8 07:55 USDCAD-2013-07.csv
-rw-rw-r--  1 fx411 fx411 121440 Nov  8 07:55 USDCAD-2013-08.csv
-rw-rw-r--  1 fx411 fx411 116840 Nov  8 07:55 USDCAD-2013-09.csv
-rw-rw-r--  1 fx411 fx411 129560 Nov  8 07:55 USDCAD-2013-10.csv
-rw-rw-r--  1 fx411 fx411 112580 Nov  8 07:55 USDCAD-2013-11.csv
-rw-rw-r--  1 fx411 fx411 120460 Nov  8 07:55 USDCAD-2013-12.csv
-rw-rw-r--  1 fx411 fx411 124760 Nov  8 07:56 USDCAD-2014-01.csv
-rw-rw-r--  1 fx411 fx411 113260 Nov  8 07:56 USDCAD-2014-02.csv
-rw-rw-r--  1 fx411 fx411 117860 Nov  8 07:56 USDCAD-2014-03.csv
-rw-rw-r--  1 fx411 fx411 123800 Nov  8 07:56 USDCAD-2014-04.csv
-rw-rw-r--  1 fx411 fx411 123140 Nov  8 07:56 USDCAD-2014-05.csv
-rw-rw-r--  1 fx411 fx411 118060 Nov  8 07:56 USDCAD-2014-06.csv
-rw-rw-r--  1 fx411 fx411 129540 Nov  8 07:57 USDCAD-2014-07.csv
-rw-rw-r--  1 fx411 fx411 117440 Nov  8 07:57 USDCAD-2014-08.csv
-rw-rw-r--  1 fx411 fx411 119040 Nov  8 07:57 USDCAD-2014-09.csv
-rw-rw-r--  1 fx411 fx411 117420 Nov  8 07:57 USDCAD-2014-10.csv
-rw-rw-r--  1 fx411 fx411  45400 Nov  8 07:57 USDCAD-2014-11.csv
-rw-rw-r--  1 fx411 fx411  47420 Nov  8 07:58 USDCAD-2014-12.csv
-rw-rw-r--  1 fx411 fx411 119200 Nov  8 07:58 USDCAD-2015-01.csv
-rw-rw-r--  1 fx411 fx411 113320 Nov  8 07:58 USDCAD-2015-02.csv
-rw-rw-r--  1 fx411 fx411 124140 Nov  8 07:59 USDCAD-2015-03.csv
-rw-rw-r--  1 fx411 fx411 118420 Nov  8 07:59 USDCAD-2015-04.csv
-rw-rw-r--  1 fx411 fx411 118220 Nov  8 07:59 USDCAD-2015-05.csv
-rw-rw-r--  1 fx411 fx411 123840 Nov  8 08:00 USDCAD-2015-06.csv
-rw-rw-r--  1 fx411 fx411 128880 Nov  8 08:01 USDCAD-2015-07.csv
-rw-rw-r--  1 fx411 fx411 118100 Nov  8 08:02 USDCAD-2015-08.csv
-rw-rw-r--  1 fx411 fx411 123860 Nov  8 08:02 USDCAD-2015-09.csv
-rw-rw-r--  1 fx411 fx411 123120 Nov  8 08:03 USDCAD-2015-10.csv
-rw-rw-r--  1 fx411 fx411 119040 Nov  8 08:03 USDCAD-2015-11.csv
-rw-rw-r--  1 fx411 fx411 126180 Nov  8 08:04 USDCAD-2015-12.csv
-rw-rw-r--  1 fx411 fx411 113040 Nov  8 08:05 USDCAD-2016-01.csv
-rw-rw-r--  1 fx411 fx411 118560 Nov  8 08:06 USDCAD-2016-02.csv
-rw-rw-r--  1 fx411 fx411 129520 Nov  8 08:06 USDCAD-2016-03.csv
-rw-rw-r--  1 fx411 fx411 117420 Nov  8 08:08 USDCAD-2016-04.csv
-rw-rw-r--  1 fx411 fx411 123860 Nov  8 08:11 USDCAD-2016-05.csv
-rw-rw-r--  1 fx411 fx411 123800 Nov  8 08:14 USDCAD-2016-06.csv
-rw-rw-r--  1 fx411 fx411 117360 Nov  8 08:16 USDCAD-2016-07.csv
-rw-rw-r--  1 fx411 fx411 129600 Nov  8 08:18 USDCAD-2016-08.csv
-rw-rw-r--  1 fx411 fx411 123120 Nov  8 08:21 USDCAD-2016-09.csv
-rw-rw-r--  1 fx411 fx411 118020 Nov  8 08:23 USDCAD-2016-10.csv
-rw-rw-r--  1 fx411 fx411 120792 Nov  8 08:23 USDJPY-2009-05.csv
-rw-rw-r--  1 fx411 fx411 114702 Nov  8 08:23 USDJPY-2009-06.csv
-rw-rw-r--  1 fx411 fx411 138138 Nov  8 08:24 USDJPY-2009-07.csv
-rw-rw-r--  1 fx411 fx411 126840 Nov  8 08:24 USDJPY-2009-08.csv
-rw-rw-r--  1 fx411 fx411 120666 Nov  8 08:25 USDJPY-2009-09.csv
-rw-rw-r--  1 fx411 fx411 132132 Nov  8 08:25 USDJPY-2009-10.csv
-rw-rw-r--  1 fx411 fx411 121296 Nov  8 08:25 USDJPY-2009-11.csv
-rw-rw-r--  1 fx411 fx411 135219 Nov  8 08:25 USDJPY-2009-12.csv
-rw-rw-r--  1 fx411 fx411 122157 Nov  8 08:26 USDJPY-2010-01.csv
-rw-rw-r--  1 fx411 fx411  85050 Nov  8 08:26 USDJPY-2010-02.csv
-rw-rw-r--  1 fx411 fx411 133245 Nov  8 08:26 USDJPY-2010-03.csv
-rw-rw-r--  1 fx411 fx411 131796 Nov  8 08:26 USDJPY-2010-04.csv
-rw-rw-r--  1 fx411 fx411 127575 Nov  8 08:27 USDJPY-2010-05.csv
-rw-rw-r--  1 fx411 fx411 123270 Nov  8 08:28 USDJPY-2010-06.csv
-rw-rw-r--  1 fx411 fx411 132132 Nov  8 08:28 USDJPY-2010-07.csv
-rw-rw-r--  1 fx411 fx411 133644 Nov  8 08:28 USDJPY-2010-08.csv
-rw-rw-r--  1 fx411 fx411 132951 Nov  8 08:29 USDJPY-2010-09.csv
-rw-rw-r--  1 fx411 fx411 126924 Nov  8 08:29 USDJPY-2010-10.csv
-rw-rw-r--  1 fx411 fx411 133224 Nov  8 08:29 USDJPY-2010-11.csv
-rw-rw-r--  1 fx411 fx411 139062 Nov  8 08:29 USDJPY-2010-12.csv
-rw-rw-r--  1 fx411 fx411 112329 Nov  8 08:30 USDJPY-2011-01.csv
-rw-rw-r--  1 fx411 fx411 121611 Nov  8 08:30 USDJPY-2011-02.csv
-rw-rw-r--  1 fx411 fx411 140154 Nov  8 08:30 USDJPY-2011-03.csv
-rw-rw-r--  1 fx411 fx411 126147 Nov  8 08:31 USDJPY-2011-04.csv
-rw-rw-r--  1 fx411 fx411 121275 Nov  8 08:31 USDJPY-2011-05.csv
-rw-rw-r--  1 fx411 fx411 126840 Nov  8 08:32 USDJPY-2011-06.csv
-rw-rw-r--  1 fx411 fx411 126021 Nov  8 08:32 USDJPY-2011-07.csv
-rw-rw-r--  1 fx411 fx411 120771 Nov  8 08:33 USDJPY-2011-08.csv
-rw-rw-r--  1 fx411 fx411 132195 Nov  8 08:33 USDJPY-2011-09.csv
-rw-rw-r--  1 fx411 fx411 126924 Nov  8 08:34 USDJPY-2011-10.csv
-rw-rw-r--  1 fx411 fx411 133833 Nov  8 08:34 USDJPY-2011-11.csv
-rw-rw-r--  1 fx411 fx411 133371 Nov  8 08:34 USDJPY-2011-12.csv
-rw-rw-r--  1 fx411 fx411 124467 Nov  8 08:35 USDJPY-2012-01.csv
-rw-rw-r--  1 fx411 fx411 123207 Nov  8 08:35 USDJPY-2012-02.csv
-rw-rw-r--  1 fx411 fx411 132531 Nov  8 08:36 USDJPY-2012-03.csv
-rw-rw-r--  1 fx411 fx411 127449 Nov  8 08:36 USDJPY-2012-04.csv
-rw-rw-r--  1 fx411 fx411 134127 Nov  8 08:37 USDJPY-2012-05.csv
-rw-rw-r--  1 fx411 fx411 116466 Nov  8 08:38 USDJPY-2012-06.csv
-rw-rw-r--  1 fx411 fx411 133539 Nov  8 08:38 USDJPY-2012-07.csv
-rw-rw-r--  1 fx411 fx411 134442 Nov  8 08:39 USDJPY-2012-08.csv
-rw-rw-r--  1 fx411 fx411 117915 Nov  8 08:39 USDJPY-2012-09.csv
-rw-rw-r--  1 fx411 fx411 135912 Nov  8 08:40 USDJPY-2012-10.csv
-rw-rw-r--  1 fx411 fx411 130305 Nov  8 08:40 USDJPY-2012-11.csv
-rw-rw-r--  1 fx411 fx411 121086 Nov  8 08:41 USDJPY-2012-12.csv
-rw-rw-r--  1 fx411 fx411 125559 Nov  8 08:42 USDJPY-2013-01.csv
-rw-rw-r--  1 fx411 fx411 112959 Nov  8 08:44 USDJPY-2013-02.csv
-rw-rw-r--  1 fx411 fx411 122262 Nov  8 08:46 USDJPY-2013-03.csv
-rw-rw-r--  1 fx411 fx411 130053 Nov  8 08:48 USDJPY-2013-04.csv
-rw-rw-r--  1 fx411 fx411 134317 Nov  8 08:50 USDJPY-2013-05.csv
-rw-rw-r--  1 fx411 fx411 111300 Nov  8 08:52 USDJPY-2013-06.csv
-rw-rw-r--  1 fx411 fx411 130714 Nov  8 08:54 USDJPY-2013-07.csv
-rw-rw-r--  1 fx411 fx411 127512 Nov  8 08:55 USDJPY-2013-08.csv
-rw-rw-r--  1 fx411 fx411 123205 Nov  8 08:55 USDJPY-2013-09.csv
-rw-rw-r--  1 fx411 fx411 136059 Nov  8 08:56 USDJPY-2013-10.csv
-rw-rw-r--  1 fx411 fx411 126989 Nov  8 08:56 USDJPY-2013-11.csv
-rw-rw-r--  1 fx411 fx411 133210 Nov  8 08:56 USDJPY-2013-12.csv
-rw-rw-r--  1 fx411 fx411 137324 Nov  8 08:57 USDJPY-2014-01.csv
-rw-rw-r--  1 fx411 fx411 124674 Nov  8 08:57 USDJPY-2014-02.csv
-rw-rw-r--  1 fx411 fx411 129690 Nov  8 08:57 USDJPY-2014-03.csv
-rw-rw-r--  1 fx411 fx411 136268 Nov  8 08:58 USDJPY-2014-04.csv
-rw-rw-r--  1 fx411 fx411 135520 Nov  8 08:58 USDJPY-2014-05.csv
-rw-rw-r--  1 fx411 fx411 129910 Nov  8 08:58 USDJPY-2014-06.csv
-rw-rw-r--  1 fx411 fx411 142604 Nov  8 08:59 USDJPY-2014-07.csv
-rw-rw-r--  1 fx411 fx411 129162 Nov  8 08:59 USDJPY-2014-08.csv
-rw-rw-r--  1 fx411 fx411 130988 Nov  8 08:59 USDJPY-2014-09.csv
-rw-rw-r--  1 fx411 fx411 136290 Nov  8 09:00 USDJPY-2014-10.csv
-rw-rw-r--  1 fx411 fx411 112068 Nov  8 09:00 USDJPY-2014-11.csv
-rw-rw-r--  1 fx411 fx411 116270 Nov  8 09:01 USDJPY-2014-12.csv
-rw-rw-r--  1 fx411 fx411 131120 Nov  8 09:01 USDJPY-2015-01.csv
-rw-rw-r--  1 fx411 fx411 124674 Nov  8 09:02 USDJPY-2015-02.csv
-rw-rw-r--  1 fx411 fx411 136576 Nov  8 09:02 USDJPY-2015-03.csv
-rw-rw-r--  1 fx411 fx411 130262 Nov  8 09:03 USDJPY-2015-04.csv
-rw-rw-r--  1 fx411 fx411 130042 Nov  8 09:03 USDJPY-2015-05.csv
-rw-rw-r--  1 fx411 fx411 136202 Nov  8 09:04 USDJPY-2015-06.csv
-rw-rw-r--  1 fx411 fx411 141768 Nov  8 09:05 USDJPY-2015-07.csv
-rw-rw-r--  1 fx411 fx411 129910 Nov  8 09:05 USDJPY-2015-08.csv
-rw-rw-r--  1 fx411 fx411 136246 Nov  8 09:06 USDJPY-2015-09.csv
-rw-rw-r--  1 fx411 fx411 135432 Nov  8 09:06 USDJPY-2015-10.csv
-rw-rw-r--  1 fx411 fx411 130944 Nov  8 09:07 USDJPY-2015-11.csv
-rw-rw-r--  1 fx411 fx411 139502 Nov  8 09:07 USDJPY-2015-12.csv
-rw-rw-r--  1 fx411 fx411 124344 Nov  8 09:08 USDJPY-2016-01.csv
-rw-rw-r--  1 fx411 fx411 130416 Nov  8 09:09 USDJPY-2016-02.csv
-rw-rw-r--  1 fx411 fx411 142560 Nov  8 09:09 USDJPY-2016-03.csv
-rw-rw-r--  1 fx411 fx411 129140 Nov  8 09:11 USDJPY-2016-04.csv
-rw-rw-r--  1 fx411 fx411 136224 Nov  8 09:14 USDJPY-2016-05.csv
-rw-rw-r--  1 fx411 fx411 136180 Nov  8 09:17 USDJPY-2016-06.csv
-rw-rw-r--  1 fx411 fx411 129096 Nov  8 09:20 USDJPY-2016-07.csv
-rw-rw-r--  1 fx411 fx411 142392 Nov  8 09:22 USDJPY-2016-08.csv
-rw-rw-r--  1 fx411 fx411 135432 Nov  8 09:25 USDJPY-2016-09.csv
-rw-rw-r--  1 fx411 fx411 129822 Nov  8 09:27 USDJPY-2016-10.csv
fx411@ub16aug:~\$
fx411@ub16aug:~\$
fx411@ub16aug:~\$
``````

I used tail to look at the very last file.

``````
fx411@ub16aug:~\$
fx411@ub16aug:~\$
fx411@ub16aug:~\$ cd csv
fx411@ub16aug:~/csv\$
fx411@ub16aug:~/csv\$ tail USDJPY-2016-10.csv
1477984200,104.845476
1477984500,104.850385
1477984800,104.882016
1477985100,104.867101
1477985400,104.863535
1477985700,104.838040
1477986000,104.832732
1477986300,104.839901
1477986600,104.827527
1477986900,104.818039
fx411@ub16aug:~/csv\$
fx411@ub16aug:~/csv\$
fx411@ub16aug:~/csv\$

fx411@ub16aug:~/csv\$
fx411@ub16aug:~/csv\$ date -u --date=@1477986900
Tue Nov  1 07:55:00 UTC 2016
fx411@ub16aug:~/csv\$
fx411@ub16aug:~/csv\$
``````

I can see that the last observation has a timestamp of Tue Nov 1 07:55:00 UTC 2016.

The mean-price for USDJPY at that 5 minute boundry was 104.818039.

After I used agg_zip.py to fill ~fx411/csv with 54 MB of CSV files, I aggregated further.

The script I used to do this second phase of aggregation is listed below:

``````
# agg_csv.py

# This script should aggregate CSV files into one file for each pair.
import glob
import os
import pandas as pd
for pair_s in pairs_l:
fn_l = glob.glob("../csv/"+pair_s+"-20*csv")
# For this pair I should sort and make uniq and output to single file
# inspiration:
# sort -u ../csv/AUDUSD-20*csv > AUDUSD.csv
if len(fn_l) > 0 :
os.system("sort -u ../csv/"+pair_s+"-20*csv > ../csv/"+pair_s+"0.csv")
p0_df = pd.read_csv("../csv/"+pair_s+"0.csv", names=['ts','cp'])
# I should use groupby to de-dup and then write to CSV:
p1_df = p0_df.groupby(['ts']).cp.mean()
p1_df.to_csv("../csv/"+pair_s+".csv", float_format='%4.6f')

'bye'
``````

The above script needed about 20 seconds to finish.

The output was one file for each pair:

``````
fx411@ub16aug:~/csv\$
fx411@ub16aug:~/csv\$ ls -la ??????.csv
-rw-rw-r-- 1 fx411 fx411 10758320 Nov  8 15:56 AUDUSD.csv
-rw-rw-r-- 1 fx411 fx411 10794620 Nov  8 15:56 EURUSD.csv
-rw-rw-r-- 1 fx411 fx411 10795860 Nov  8 15:56 GBPUSD.csv
-rw-rw-r-- 1 fx411 fx411 10719260 Nov  8 15:56 USDCAD.csv
-rw-rw-r-- 1 fx411 fx411 11596881 Nov  8 15:56 USDJPY.csv
fx411@ub16aug:~/csv\$
fx411@ub16aug:~/csv\$
fx411@ub16aug:~/csv\$
``````

I considered the above files suitable for the next step, the feature creation step.