Hyperopt loss going wrong way ?

[taz4hvn]

Describe your environment

• Operating system: ____ Manjaro
• Python Version: _____ Python 3.11.8
• CCXT version: _____ ccxt==4.2.76
• Freqtrade Version: ____ freqtrade 2024.4

Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

Describe the problem:

Hi. It seems loss function score is broken somehow (maybe smthg to do with its sign).
Example below (as on 2024-05-07) says #5 is best while obviously it should be #10.
Regards.

Steps to reproduce:

1. command used:
   docker-compose run --rm freqtrade hyperopt --disable-param-export --config ./user_data/config_binanceWEB.json --strategy TAZ_15m_binance000 --timerange=20230901-20240501 --spaces buy --hyperopt-loss ProfitDrawDownHyperOptLoss --epoch 1000 --timeframe 15m --timeframe-detail 1m --j 1

Observed Results:

+--------+---------+----------+--------------------------+--------------+---------------------------+----------------+--------------+---------------------------+
|   Best |   Epoch |   Trades |    Win  Draw  Loss  Win% |   Avg profit |                    Profit |   Avg duration |    Objective |       Max Drawdown (Acct) |
|--------+---------+----------+--------------------------+--------------+---------------------------+----------------+--------------+---------------------------|
| * Best |   1/135 |        4 |      3     0     1  75.0 |        1.59% |        30.785     (3.08%) |        1:35:00 |    -29.70234 |         3.559     (0.35%) |
| *      |   2/135 |       11 |      8     0     3  72.7 |       -0.94% |      -221.333   (-22.13%) |        0:24:00 |    247.98383 |       145.339    (15.98%) |
| *      |   3/135 |       51 |     42     0     9  82.4 |        0.43% |      -315.876   (-31.59%) |        0:54:00 |     87.71371 |       417.773    (40.04%) |
| *      |   4/135 |        1 |      1     0     0   100 |        2.29% |         3.795     (0.38%) |        0:00:00 |     -3.79499 |                        -- |
| * Best |   5/135 |       51 |     42     0     9  82.4 |        0.11% |      -459.923   (-45.99%) |        0:28:00 | -4,180.04668 |       510.315    (50.84%) |
| *      |   6/135 |      170 |    145     0    25  85.3 |        0.69% |      -581.544   (-58.15%) |        0:52:00 |   -100.19843 |       721.638    (68.98%) |
| *      |   7/135 |        2 |      2     0     0   100 |        2.29% |         7.604     (0.76%) |        0:00:00 |     -7.60435 |                        -- |
| *      |   8/135 |       15 |     14     0     1  93.3 |        1.78% |        64.167     (6.42%) |        1:46:00 |    -59.80514 |        56.433     (5.31%) |
| *      |   9/135 |      184 |    158     0    26  85.9 |        0.74% |      -628.566   (-62.86%) |        0:46:00 |     11.19232 |       790.266    (74.53%) |
| *      |  10/135 |      190 |    174     0    16  91.6 |        1.48% |       602.105    (60.21%) |        1:01:00 |   -586.82945 |       304.910    (16.04%) |
| *      |  11/135 |        5 |      5     0     0   100 |        2.29% |        22.971     (2.30%) |        0:03:00 |    -22.97094 |                        -- |
| *      |  12/135 |      345 |    298     0    47  86.4 |        0.94% |      -494.568   (-49.46%) |        1:21:00 |     17.80432 |       746.868    (70.59%) |
| *      |  13/135 |       44 |     37     0     7  84.1 |        0.64% |      -251.262   (-25.13%) |        0:55:00 |   -624.95887 |       352.895    (35.02%) |
| *      |  14/135 |        2 |      2     0     0   100 |        2.29% |         7.604     (0.76%) |        0:00:00 |     -7.60435 |                        -- |
| *      |  15/135 |        4 |      4     0     0   100 |        2.29% |        15.263     (1.53%) |        0:01:00 |    -15.26296 |                        -- |
| *      |  16/135 |       49 |     40     0     9  81.6 |        0.20% |      -429.174   (-42.92%) |        0:35:00 |   -901.90677 |       471.865    (46.65%) |
| *      |  17/135 |      362 |    310     0    52  85.6 |        0.79% |      -759.062   (-75.91%) |        1:10:00 | -1,031.79724 |       842.359    (82.04%) |

[xmatthias]

Well the calculation is not really ideal - that's also been reported in #9563 and #9819.

The sign is however not "really" the problem (at least not the only one) - it's the overall combination in the formula - as well as the (at least in part, ignorance of the range of values a value can reach.

While the variable says "max_drawdown_abs" - it's really a relative value that's retrieved.
it's however NOT bound to be between 0 and one - but rather, it's unbound in it's upper bound, as the account level drawdown can be > 100% (it's drawdown from the high point, not from starting balance).

So if your strategy does great, your "high" value may be 100% profit (from 1000$ to 2000$) - but then it's in a bad run and loses 1200$ (your net-loss is 200$).
What you'd get then is ~1.2 (120%).

---

for your particular result - if you simply invert the sign - you'd get #5 as "worst" result ... but #2 as best ... which is still a rather losing strategy.
#12 would be pretty high up, too - which is losing AND has a pretty high drawdown - both not very ideal situations.

---

In reality / summary - i think the way this loss function combines the values is not ideal - and I'm more than happy to get a PR that sees it improved.

The (simple, from a user perspective) alternative is to pick a different loss function - which there's plenty of - and all try to search for different "improvements".

[taz4hvn]

Well, thanks for the comprehensive answer. I guess I'll stick with OnlyProfitHyperOptLoss and manually discard frightening drawdowns for now.

[taz4hvn]

Hi. I tested 2024.5-dev-ea27a1ec implementing new formula proposed by #10221. Now objective is positive so freqtrade actually tries to improve the worst winning strategy. Appears to me a simple minus sign or min/max swap is needed.

Example:

+--------+---------+----------+--------------------------+--------------+---------------------------+----------------+---------------+---------------------------+
| Best | Epoch | Trades | Win Draw Loss Win% | Avg profit | Profit | Avg duration | Objective | Max Drawdown (Acct) |
|--------+---------+----------+--------------------------+--------------+---------------------------+----------------+---------------+---------------------------|
| * | 1/76 | 734 | 602 0 132 82.0 | 3.95% | 19556.873 (1,955.69%) | 3:55:00 | 38,257.15248 | 4139.799 (22.17%) |
| * | 2/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 3/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 4/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 5/76 | 1012 | 791 0 221 78.2 | 3.21% | 20645.571 (2,064.56%) | 4:19:00 | 66,862.57108 | 9622.858 (31.79%) |
| * | 6/76 | 601 | 498 0 103 82.9 | 4.39% | 14163.363 (1,416.34%) | 4:01:00 | 28,566.10630 | 3043.671 (22.63%) |
| * | 7/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 8/76 | 599 | 495 0 104 82.6 | 4.31% | 12034.538 (1,203.45%) | 4:02:00 | 24,272.50466 | 2627.944 (22.63%) |
| * | 9/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 10/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 11/76 | 629 | 517 0 112 82.2 | 4.30% | 17080.542 (1,708.05%) | 4:07:00 | 34,070.72703 | 3684.872 (22.46%) |
| * | 12/76 | 744 | 609 0 135 81.9 | 3.87% | 19631.124 (1,963.11%) | 3:52:00 | 38,402.39922 | 4246.497 (22.17%) |
| * | 13/76 | 684 | 558 0 126 81.6 | 4.11% | 16423.782 (1,642.38%) | 3:59:00 | 33,935.62496 | 3942.400 (23.00%) |
| * | 14/76 | 629 | 522 0 107 83.0 | 4.34% | 15421.954 (1,542.20%) | 3:59:00 | 32,985.99514 | 3466.805 (23.54%) |
| * | 15/76 | 602 | 498 0 104 82.7 | 4.35% | 13823.615 (1,382.36%) | 4:01:00 | 28,923.84756 | 3085.955 (23.19%) |
| * | 16/76 | 726 | 594 0 132 81.8 | 3.91% | 19059.908 (1,905.99%) | 3:45:00 | 41,485.29107 | 4725.894 (23.82%) |
| * | 17/76 | 598 | 495 0 103 82.8 | 4.35% | 13328.017 (1,332.80%) | 4:02:00 | 27,886.88265 | 2982.782 (23.19%) |
| * | 18/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 19/76 | 631 | 523 0 108 82.9 | 4.24% | 14924.618 (1,492.46%) | 3:55:00 | 31,303.92590 | 3679.231 (23.23%) |
| * | 20/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 21/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 22/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 23/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 24/76 | 599 | 496 0 103 82.8 | 4.37% | 13702.669 (1,370.27%) | 4:02:00 | 28,670.80167 | 3025.407 (23.19%) |
| * | 25/76 | 606 | 500 0 106 82.5 | 4.32% | 12287.715 (1,228.77%) | 4:07:00 | 24,783.12308 | 2652.318 (22.63%) |
| * | 26/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 27/76 | 609 | 502 0 107 82.4 | 4.34% | 14710.133 (1,471.01%) | 4:01:00 | 30,778.73363 | 3239.703 (23.19%) |
| * | 28/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 29/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| * | 30/76 | 1129 | 896 0 233 79.4 | 3.08% | 36618.966 (3,661.90%) | 3:50:00 | 130,356.35571 | 18148.476 (34.20%) |
| | 31/76 | 795 | 651 0 144 81.9 | 3.55% | 15774.820 (1,577.48%) | 3:33:00 | 33,186.70003 | 4172.133 (23.28%) |
| | 32/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| | 33/76 | 599 | 496 0 103 82.8 | 4.35% | 13413.411 (1,341.34%) | 4:01:00 | 27,053.54215 | 2905.942 (22.63%) |
| | 34/76 | 598 | 494 0 104 82.6 | 4.31% | 11937.076 (1,193.71%) | 4:02:00 | 24,976.56314 | 2693.220 (23.19%) |
| | 35/76 | 601 | 497 0 104 82.7 | 4.33% | 12515.382 (1,251.54%) | 4:01:00 | 26,186.56351 | 2801.211 (23.19%) |
| | 36/76 | 601 | 497 0 104 82.7 | 4.34% | 12499.879 (1,249.99%) | 4:01:00 | 26,154.14337 | 2810.383 (23.19%) |
| | 37/76 | 597 | 493 0 104 82.6 | 4.30% | 11652.598 (1,165.26%) | 4:02:00 | 24,381.33363 | 2633.998 (23.19%) |
| | 38/76 | 600 | 496 0 104 82.7 | 4.32% | 12306.063 (1,230.61%) | 4:01:00 | 24,820.12574 | 2667.735 (22.63%) |
| | 39/76 | 600 | 496 0 104 82.7 | 4.33% | 12377.952 (1,237.80%) | 4:01:00 | 24,965.13560 | 2682.150 (22.63%) |
| | 40/76 | 607 | 502 0 105 82.7 | 4.36% | 13275.814 (1,327.58%) | 3:59:00 | 26,928.87442 | 2863.620 (22.71%) |
| | 41/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| | 42/76 | 599 | 495 0 104 82.6 | 4.30% | 12156.656 (1,215.67%) | 4:02:00 | 25,435.98772 | 2738.930 (23.19%) |
| | 43/76 | 601 | 497 0 104 82.7 | 4.31% | 12291.424 (1,229.14%) | 4:01:00 | 25,858.44901 | 2770.979 (23.28%) |
| | 44/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| | 45/76 | 603 | 499 0 104 82.8 | 4.34% | 12759.322 (1,275.93%) | 4:00:00 | 25,881.21752 | 2772.232 (22.71%) |
| | 46/76 | 598 | 494 0 104 82.6 | 4.29% | 11750.330 (1,175.03%) | 4:02:00 | 24,585.83273 | 2654.345 (23.19%) |
| | 47/76 | 599 | 495 0 104 82.6 | 4.30% | 12039.121 (1,203.91%) | 4:02:00 | 25,190.06864 | 2714.462 (23.19%) |
| | 48/76 | 600 | 496 0 104 82.7 | 4.31% | 12795.640 (1,279.56%) | 4:01:00 | 25,807.56056 | 2781.391 (22.63%) |
| | 49/76 | 599 | 495 0 104 82.6 | 4.30% | 12098.794 (1,209.88%) | 4:02:00 | 25,314.91856 | 2726.884 (23.19%) |
| | 50/76 | 605 | 497 0 108 82.1 | 4.28% | 12721.381 (1,272.14%) | 4:00:00 | 25,804.24734 | 2855.773 (22.71%) |
| | 51/76 | 602 | 498 0 104 82.7 | 4.35% | 12664.180 (1,266.42%) | 4:00:00 | 25,542.43362 | 2739.536 (22.63%) |
| | 52/76 | 604 | 500 0 104 82.8 | 4.32% | 13275.071 (1,327.51%) | 4:00:00 | 27,927.83235 | 2986.077 (23.28%) |
| | 53/76 | 604 | 499 0 105 82.6 | 4.34% | 12950.335 (1,295.03%) | 3:58:00 | 26,268.66078 | 2801.277 (22.71%) |
| | 54/76 | 604 | 499 0 105 82.6 | 4.32% | 12934.186 (1,293.42%) | 4:00:00 | 26,235.90012 | 2923.139 (22.71%) |
| | 55/76 | 602 | 498 0 104 82.7 | 4.33% | 12849.671 (1,284.97%) | 4:00:00 | 25,916.52942 | 2776.723 (22.63%) |
| | 56/76 | 603 | 499 0 104 82.8 | 4.31% | 13075.987 (1,307.60%) | 4:01:00 | 26,372.97766 | 2850.563 (22.63%) |
| | 57/76 | 599 | 495 0 104 82.6 | 4.30% | 12823.897 (1,282.39%) | 4:02:00 | 26,832.09730 | 2877.837 (23.19%) |
| | 58/76 | 602 | 498 0 104 82.7 | 4.31% | 12975.882 (1,297.59%) | 4:01:00 | 26,171.09430 | 2802.028 (22.63%) |
| | 59/76 | 604 | 500 0 104 82.8 | 4.36% | 13023.474 (1,302.35%) | 4:00:00 | 26,267.07810 | 2811.570 (22.63%) |
| | 60/76 | 605 | 500 0 105 82.6 | 4.35% | 13089.809 (1,308.98%) | 4:00:00 | 26,400.87714 | 2812.422 (22.63%) |
| | 61/76 | 602 | 498 0 104 82.7 | 4.31% | 12975.882 (1,297.59%) | 4:01:00 | 26,171.09430 | 2802.028 (22.63%) |
| | 62/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| | 63/76 | 607 | 502 0 105 82.7 | 4.30% | 12297.110 (1,229.71%) | 4:00:00 | 24,943.64966 | 2781.250 (22.71%) |
| | 64/76 | 613 | 509 0 104 83.0 | 4.34% | 14645.958 (1,464.60%) | 3:57:00 | 30,811.86288 | 3223.874 (23.28%) |
| | 65/76 | 602 | 498 0 104 82.7 | 4.32% | 13356.179 (1,335.62%) | 4:01:00 | 26,938.09286 | 2878.272 (22.63%) |
| | 66/76 | 604 | 500 0 104 82.8 | 4.36% | 13023.474 (1,302.35%) | 4:00:00 | 26,267.07810 | 2811.570 (22.63%) |
| | 67/76 | 616 | 511 0 105 83.0 | 4.30% | 14003.897 (1,400.39%) | 3:57:00 | 28,405.72266 | 3090.325 (22.71%) |
| | 68/76 | 598 | 495 0 103 82.8 | 4.35% | 13307.999 (1,330.80%) | 4:02:00 | 27,845.00588 | 2978.616 (23.19%) |
| | 69/76 | 599 | 495 0 104 82.6 | 4.31% | 12456.486 (1,245.65%) | 4:02:00 | 24,847.12094 | 2679.339 (22.46%) |
| | 70/76 | 605 | 499 0 106 82.5 | 4.29% | 12492.518 (1,249.25%) | 4:06:00 | 25,349.64940 | 2726.529 (22.72%) |
| | 71/76 | 600 | 496 0 104 82.7 | 4.31% | 12591.008 (1,259.10%) | 4:02:00 | 25,115.43947 | 2706.122 (22.46%) |
| | 72/76 | 611 | 504 0 107 82.5 | 4.25% | 13014.801 (1,301.48%) | 4:05:00 | 26,409.46781 | 2832.072 (22.72%) |
| | 73/76 | 603 | 499 0 104 82.8 | 4.33% | 13473.301 (1,347.33%) | 4:01:00 | 28,190.86992 | 3013.028 (23.19%) |
| | 74/76 | 613 | 508 0 105 82.9 | 4.29% | 13777.477 (1,377.75%) | 4:01:00 | 27,787.81039 | 2961.171 (22.63%) |
| | 75/76 | 598 | 494 0 104 82.6 | 4.31% | 11937.076 (1,193.71%) | 4:02:00 | 24,976.56314 | 2693.220 (23.19%) |
| | 76/76 | 610 | 503 0 107 82.5 | 4.26% | 12962.665 (1,296.27%) | 4:01:00 | 26,144.41571 | 2903.681 (22.63%) |
+--------+---------+----------+--------------------------+--------------+---------------------------+----------------+---------------+---------------------------+

PS: I dunno if commenting on this post was the correct procedure ?

[xmatthias]

It's interesting insofar as my experience was different - and results DID improve quite significantly, preferring better results.

[xmatthias]

Are you sure that you're using the correct loss function (ProfitDrawDownHyperOptLoss)

I'm asking this because your objective's do not align with that calculation.
Using your very first result - the calculation happens with the following parameters

DRAWDOWN_MULT = 0.075
total_profit  = 19556
rel_drawdown = 22.17 / 100

-1 * (total_profit - (rel_drawdown * DRAWDOWN_MULT * total_profit))

and results in an objective of -19230.83261.

Your objective on the other hand is 38,257.15248 - which is something completely different - and i don't really see how this can happen with the given values.

[taz4hvn]

hi,

FIRST RUN

I did a new test, since previous is lost.

COMMAND:
docker-compose run --rm freqtrade hyperopt --disable-param-export --config ./user_data/config_kucoinWEB.json --strategy TAZ_kucoin320b --timerange=20231201-20240522 --spaces buy --hyperopt-loss ProfitDrawDownHyperOptLoss --epoch 1000 --timeframe 5m --timeframe-detail 1m --j 1

FREQTRADE:
freqtrade docker-2024.5-dev-ea27a1ec

OUTPUT:

+--------+---------+----------+--------------------------+--------------+---------------------------+----------------+--------------+---------------------------+
|   Best |   Epoch |   Trades |    Win  Draw  Loss  Win% |   Avg profit |                    Profit |   Avg duration |    Objective |       Max Drawdown (Acct) |
|--------+---------+----------+--------------------------+--------------+---------------------------+----------------+--------------+---------------------------|
| *      |     1/2 |      985 |    806     0   179  81.8 |        3.57% |    39529.212  (3,952.92%) |        3:39:00 | 76,461.97295 |      7606.080    (22.01%) |
| *      |     2/2 |     1095 |    878     0   217  80.2 |        3.40% |    49276.039  (4,927.60%) |        4:05:00 | 99,701.15153 |     10293.676    (22.67%) |
+--------+---------+----------+--------------------------+--------------+---------------------------+----------------+--------------+---------------------------+

CALCULATION:
for epoch 1: -1x(39529.212-(0.2201x0.075x39529.212)=-38876 same weird behavior you pointed.

SECOND RUN

after a pull

FREQTRADE:
freqtrade docker-2024.5-dev-2c740059

OUTPUT:

+--------+---------+----------+--------------------------+--------------+---------------------------+----------------+--------------+---------------------------+
|   Best |   Epoch |   Trades |    Win  Draw  Loss  Win% |   Avg profit |                    Profit |   Avg duration |    Objective |       Max Drawdown (Acct) |
|--------+---------+----------+--------------------------+--------------+---------------------------+----------------+--------------+---------------------------|
| *      |     1/2 |      986 |    807     0   179  81.8 |        3.58% |    39887.289  (3,988.73%) |        3:39:00 | 77,154.60589 |      7673.281    (22.01%) |
| *      |     2/2 |     1020 |    836     0   184  82.0 |        3.59% |    48260.136  (4,826.01%) |        3:38:00 | 93,350.30176 |      9424.433    (22.01%) |
+--------+---------+----------+--------------------------+--------------+---------------------------+----------------+--------------+---------------------------+

same positive Objective values with this build.

[taz4hvn]

in:
https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/optimize/hyperopt_loss/hyperopt_loss_profit_drawdown.py
i can read:
return -1 * (total_profit - (relative_account_drawdown * total_profit) / DRAWDOWN_MULT)
when applied to my last first run gives 76475 for epoch1 and 99669 for epoch2, pretty close (but not equal ?) to what I got.
Seems formula is not what you think it is ...

PS: coherent with 480477d

[xmatthias]

You're right, the one i used in my calculations was not the very last try.
i've updated this now to how i ACTUALLY wanted to commit it - so it should work better now.