Building A Trading System

[userque]

Did:

• Added dynamic SMA crossover with the underlying.
• Gave algo the ability to mix all or some of the indicators to generate a signal.
• Fixed bug.

Todo:

• Make all indicators dynamic
• Give algo ability to choose which indicator to follow dynamically (One day, follow this indicator, next day, follow that indicator)
• Modify Trend indicator to represent breakouts to the upside and downside.
• Convert SMA to EMA

Latest version has a final balance of $6200 ($1000 initial) using just the 20 minute algo setting.

Full Data:
D A T A A C C - Pastebin.com

Pastebin post expires in about a month.

Note:

Not sure if I mentioned that the longer the algo runs, the more likely it'll find a better solution. In this case, there is a cumulative effect. I can run the algo everyday, for varying lengths of times, and still possibly see improvement. This is something I'd do once the final version is ready, not during this testing phase.

Edit:

One of the parameters was at its maximum value as constrained by the algo. I increased the maximum value constraint for that parameter.

 

[userque]

Better post this before I forget. Still just casually and quickly testing things.

Forgot to mention that I had the Solver algo set to shut down after 10 minutes without any improvement. I can probably achieve better results using a longer setting. But I usually use a shorter setting for comparing systems etc. I did push it up to 20 minutes though.

As I posted, the trend system clocked in at $2700 with $1000 initial. I added a delay feature to it.

The delay parameter allowed the algo to choose a number of delays to delay issuing the signal, after the actual signal was generated. Reasoning:

Many times, when a higher high occurs, there is first a pullback, prior to a continuation of the upward thrust. With this change, there was a little improvement...to 3100 I believe.
__________________

There are a few traditional indicators I feel have promise (non lagging indicators): Stop and Reverse, Breakouts, and Channel fades. The others may work under the right conditions, not sure. I'll test some here.

I coded the SAR, it did better than the Trend alone: 3900. That equates to doubling your money twice over 16 years. Respectable I guess. I combined the SAR with the Trend and bumped the results up to 4100.

The SAR works by setting a stop-loss that the algo determines, and setting a "profit-reverse," also determined by the algo.

I primed it by making it go long on the first day of my historical data, sometime in Oct, 1999. Then, it sets the stop-loss price and the profit-reverse price, and signal for all subsequent dates--the backtest. I also added a "time reverse" parameter to the algo.

The "time reverse" forces the system to reverse position after a certain number of days as determined by the also.

The system will reverse, then reset the stop-loss and reverse values if the stop-loss, profit-reverse, or time-reverse targets are hit.

The SAR along with the TREND together did 4600 (after I bumped the algo run time without improvements to 20 minutes). Keep in mind, this system is only going LONG and doesn't include G- or F-Fund trades.
_____________

Prior to doing the above, I did test a SMA crossover with another SMA and that configuration did poorly. Next, I will test a SMA crossover with the underlying where the SMA parameter is determined dynamically by a custom volatility indicator I'll create. The algo will figure out all of the parameter settings for us, of course.
_____________

I've tested a lot of these things before, not all of them though. None of them came close to my pattern matching systems, which I currently use and develop. Again, I've always been curious as to how these non-pattern systems would perform if I built them now...knowing what I know now. JTH motivated me, so here we are.

 

[userque]

Part 1: I'm glad you were able to configure the 1-day delay, this was very difficult for me on the Ninja platform, due to my limited coding skills, I had scoured for weeks before stumbling on a rouge post that led to the config. It made me so happy to perform TSP-style backtesting using a 1-day end of price delay. It also opened my eyes to just how much a 1-day delay can change the results of the returns. Those scans sucked hours of processing time, and choosing different time frames could produce different results. I actually burned up my PC doing that, the constant heat toasted that sucker! Anyways, in the end I was unable to find a major indicator that provided the magic bullet, and it's those hundreds of hours of scanning that formed my belief that indicators are junk (it's the person's use of the indicator that's more important).

Part 2: I'll need some time to digest the entire post, there's much to consider and it's a bit overwhelming. I understand RMI's perspective on a longer timeframe, I was expecting to encounter this because the 2-IFT 1-Day-delay buy price really jacks things up. I would like to know more about what you thought of my theory that when flat prices happen, they usually will take the direction of the previous trend. This assumption is very important to how I trade, and if I am wrong, then I need to adjust my thinking.

I've got the 13th week and the 2nd quarter blog to work on this weekend, so I'll be hella busy, thank you for working on this, I think we all are learning something from these results...Jason

Thanks Jason.

Wow...my laptop gets warm, but I think it slows itself down if it needs to. I agree about the use of indicators. I also believe a machine can "look" at more of them and make better use of them than we can, if properly coded.

Yes, dealing with the delay and 2 ift per month thing can make your brain smoke the first time you face it. Fortunately, this was not my first time, but it was still a headache.

Regarding flat prices...well...the algorithm seems to agree with you on that one! The reason I didn't "hard code" that notion in is because, you never know how all the variables will interact with each other. So rather than make assumptions, even those that seem obvious, if it's not to much trouble--I like to give the machine to ability to decide for itself, what a parameter should be.

I was shocked it didn't perform better, even though it is only going LONG. I was also surprised that it chose such a long window.

I suspect it is curve fitting to the bull market runs we've had in the past. It's ignoring the wiggles in exchange for the big returns during the bull runs. Just a guess so far. Algos tend to take the path of least resistance. You have to properly constrain them to get the proper output.

Also, these algos (Excel Solver) aren't brute-forcing a solution. Sometimes, they miss the optimal solution.

 

[JTH]

Part 1: Here's how it should work. After the close, I update the sheet with that day's data. It will then generate a signal. The IFT won't be recognized until the next day. AND, the signal/IFT won't take effect until the end of that next day! Tricky.
___________


Part 2: It determined the best window size was 87! (4 months!...as RMI hypothesized) with continuing the previous trend when no trend is detected (as JTH hypothesized was best). And it calculated it was best to base the trend on the Highs and Lows, instead of the Closing price.
___________

Part 1: I'm glad you were able to configure the 1-day delay, this was very difficult for me on the Ninja platform, due to my limited coding skills, I had scoured for weeks before stumbling on a rouge post that led to the config. It made me so happy to perform TSP-style backtesting using a 1-day end of price delay. It also opened my eyes to just how much a 1-day delay can change the results of the returns. Those scans sucked hours of processing time, and choosing different time frames could produce different results. I actually burned up my PC doing that, the constant heat toasted that sucker! Anyways, in the end I was unable to find a major indicator that provided the magic bullet, and it's those hundreds of hours of scanning that formed my belief that indicators are junk (it's the person's use of the indicator that's more important).

Part 2: I'll need some time to digest the entire post, there's much to consider and it's a bit overwhelming. I understand RMI's perspective on a longer timeframe, I was expecting to encounter this because the 2-IFT 1-Day-delay buy price really jacks things up. I would like to know more about what you thought of my theory that when flat prices happen, they usually will take the direction of the previous trend. This assumption is very important to how I trade, and if I am wrong, then I need to adjust my thinking.

I've got the 13th week and the 2nd quarter blog to work on this weekend, so I'll be hella busy, thank you for working on this, I think we all are learning something from these results...Jason

 

[userque]

Solver in action:
Video linker not working for me.
https://vimeo.com/160240439

 

[userque]

Tidied things up a little.

Added Parameters: TradesPerMonth and Starting Balance.
Added Simulated Account
Added Metrics Section
___________

Here's how it should work. After the close, I update the sheet with that day's data. It will then generate a signal. The IFT won't be recognized until the next day. AND, the signal/IFT won't take effect until the end of that next day! Tricky.
___________

I first ran Solver's genetic/evolutionary algorithm optimizer using the default settings. The best balance it was able to achieve was $2400.

I then used my custom settings and got the balance up to over $2700.

The algorithm changes the values it's told to change to achieve an optimized result. We wanted it to tweak: the window size; whether the system should go to cash, or hold its position, when no trend is detected; and whether the trend should be determined by the Closing price, or the Highs and Lows.

It determined the best window size was 87! (4 months!...as RMI hypothesized) with continuing the previous trend when no trend is detected (as JTH hypothesized was best). And it calculated it was best to base the trend on the Highs and Lows, instead of the Closing price.
___________

Many optimize just the "ending balance" with ML. I've found that you have to be more creative. Algos can tend to sacrifice many months/years of returns just for a big payoff during a particular time frame. A.K.A "curve fitting."

Instead of just optimizing (maximizing) the ending balance, use a metric like, "the worst drawdown." Have the algo 'minimize' the worst 20 (or whatever) day period...or maximize the average gain per a certain period, something like that. Imo. You could use the Sharpe Ratio, or whatever you think is best to optimize. Whatever is most important to you.
___________

Next, I'll add some metrics like the above-mentioned.
___________

Note, we haven't used any technical indicators nor stop-losses. We could even add a delay; such as buying two days after the signal, etc. Or only when there is some sort of pull back after a signal.

When you can code your own system with a 'real' programming platform, you can do whatever you can imagine! (And we haven't even wrote any computer code!)

$1700 profit from $1000 initial since 1999 is not good.

What else shall we teach this system? It needs help. It needs more tools to beat the markets.

 

[userque]

Very impressive Q and JTH. Looking forward to seeing this in action. Please keep us posted on updates.

FS

Thanks FS...will do!

 

[FogSailing]

Very impressive Q and JTH. Looking forward to seeing this in action. Please keep us posted on updates.

FS

 

[RealMoneyIssues]

...getting out late...

I think you mean.

The longer/higher...you get more delayed signals. You get in later. You get out later. A dampening effect.

tomatoes, tomatoes... all in the number you use.

 

[userque]

Spectator or idiot?


I believe that really depends on the numbers used... the longer(higher), the less whipsaw, but also the better chance of getting out early or taking too much of a lose before getting out.

...getting out late...

I think you mean.

The longer/higher...you get more delayed signals. You get in later. You get out later. A dampening effect.

 

[RealMoneyIssues]

Sorry, you don't get to hold that title.

Spectator or idiot?

As I understand it, this is a portion of the overall system, where we first determine if the market is trending, oscillating, or transitioning. This helps us to understand what tools to use, based on the conditions the markets are operating under.

As an example, you might not want to use a moving average indicator during oscillating (whipsawing) conditions because it could force you in & out of the markets prematurely.

I believe that really depends on the numbers used... the longer(higher), the less whipsaw, but also the better chance of getting out early or taking too much of a lose before getting out.

 

[userque]

This stuff sounds way too complicated for little 'ole me to grasp. I like to keep things simple like "invest in gold, get wealthy." I know visuals speak louder than words and I'm one step ahead already. See the below visual I've created for you boys:

Boom! Any questions?




Sent from my iPhone using Tapatalk

LOL...no questions

When we're finished and it's polished...it'll be simple to use...you won't have to build it from scratch like we're doing here.

Also:

Added additional parameter to determine whether TREND is based on the Close, or the H/L's. Now, on to the simulated account. After that, the machine learning.

 

[jpcavin]

This stuff sounds way too complicated for little 'ole me to grasp. I like to keep things simple like "invest in gold, get wealthy." I know visuals speak louder than words and I'm one step ahead already. See the below visual I've created for you boys:

Boom! Any questions?




Sent from my iPhone using Tapatalk

 

[userque]

I think I'll do the TREND based on H/L before doing the simulated account.

Looking way ahead, it would be interesting to build a system that tracked many 'systems'/indicators and traded based on how the best system it tracks is trading.

 

[userque]

Window at 9, Cash enabled:

 

[userque]

With the window set to 9 instead of 5:

 

[userque]

Finished our custom TREND indicator. Added a new parameter. The new parameter tells the system to treat NO TREND as either a move to CASH, or a continuation of the previous signal.

No one mentioned this, but we eventually should add an additional TREND indicator that's based on the High and/or Low of the day, instead of the Close. We can see how they compare to each other with regard to profit.

EDIT: 1 is up trend, -1/(1) is down trend, 0 is cash (0 only appears if the parameter is set for it to appear)

On to the simulated account.

(Btw, it doesn't take long to build this; I'm just nibbling at it here and there. )

 

[userque]

Interesting, I did something similar a few years back, with the T21220-23 indicators on the 4/13/26/52 high/low charts from Freestockcharts.com. I used them to identify oversold conditions, but never applied them in a way to identify all market conditions. This should be fun...

Fun...Absolutely! Like you, I started something like this a while back, but abandoned it for a precursor to my current system. I've always wondered how a ML system, giving every sort of indicator (rather than looking for patterns as I currently do), would perform; but I've never been motivated enough to build it.

Doing it publicly, while also helping others, gives me the added motivation I need.

 

[userque]

Next, I have to logically tie it all together into one TREND SIGNAL, before moving on to the simulated account.

 

[JTH]

View attachment 37704

• Parameters etc. will be in the first two columns.
• The data is SP500.
• MAX is the maximum close within the time frame defined by the parameter.
• HIHELP is a column used to help calculate the HI column.
• HI? indicates whether the close for that row/day is a local high.
• PREVHI indicates the previous local high value.
• HIERHI indicates whether the close for that row/day is a higher high, lower high, or neither. 1 is a higher high. -1/(1) is a lower high. 0 is neither.

Next, I'll repeat this for the LOW.

Then, I'll add the simulated trading account code.

Interesting, I did something similar a few years back, with the T21220-23 indicators on the 4/13/26/52 high/low charts from Freestockcharts.com. I used them to identify oversold conditions, but never applied them in a way to identify all market conditions. This should be fun...