Crypto Coarse Universe Selection is now Available

Looking forward to trying out the new feature. Keep up the great work!

Hey Louis, this is awesome, thanks! However, when I tried to clone the algorithm, I receive an error that 'CryptoCoarseFundamentalUniverse' is not defined. Any idea why this is?

Hi Dave

This function is only available in IDE 3.0, since the latest LEAN version available for IDE 2.0 doesn't include it. Please upgrade to that by navigating to Your username at top right-hand corner -> My Account -> check "QuantConnect Algorithm Environment 3.0 Beta".

Best
Louis

Louis Szeto  

I tried it for Market.Binance. It says that on 2021-01-01 the top5 symbols by dollar volume over 10million usd are the following. Is this plausible?

def UniverseSelectionFilter(self, data):
	filtered = [datum for datum in data if datum.Price >= 1.00 and datum.VolumeInUsd >=10000000 and str(datum.Symbol.Value).endswith("USDT")]
	sorted_by_volume_in_usd = sorted(filtered, key=lambda datum: datum.VolumeInUsd, reverse=True)[0:5]
	symbols=[datum.Symbol for datum in sorted_by_volume_in_usd]
	self.log(f"UNIVERSE:....{str([str(e.Value) for e in symbols])}")
	return symbols

2021-01-02 00:00:00, UNIVERSE:....['1INCHUSDT', 'DOTUSDT', 'THETAUSDT', 'UNIUSDT', 'EURUSDT'] 

Hi Louis,

I got same issue as mentioned by Santa24.

I tried your demo code using Bitfinex and Binance, with recent data like SetStartDate (2022, 5, 9), but found the Symbol list generated by UniverseSelectionFilter, sorted by VolumeInUsd, is apparently different from the list I got from Bitfinex or Binance APP.  

Hi Santa24 and Tony

Unlike statistics provided on the brokers’ websites, our data processing mechanism is based on all available trading pairs in the exchange, not limited to stablecoin pairs only. Also, our data is processed based on the last 24 hours from 00:00 UTC, but not the last 24 hours in real-time. Thus, the data will have a discrepancy.

As per the “weird USDT pairs” query, please be noted that USDT pairs could also be in form of “USDTxxx”, so you may want to call:

"USDT" in datum.Symbol.Value

Best
Louis

Yeah, something does feel odd here.

The list that Santa24 is seeing seems wrong. For example, I'd expect to see BTC/USDT high in this list.

For reference, here's a list of currency markets (pairs) ranked by trading volume: 

https://coinranking.com/markets

Hi .ekz, Santas24 and Tony

Thank you for reporting the issue. This is due to a bug in the data processor that mistakenly took the base currency as the quote currency of the pair. It is now resolved. We're sorry for the inconvenience caused.

Best
Louis

throws run time error about fundamental data

Hi Louis，

Thank you for your effort in addressing these issues. I tried again and found the issue seems still there - top USD volume crypto pair list generated by Coarse is totally different from what I got from Binance APP for the same period. 

In addition, I also found:

• If using your demo code, only change “filtered = [datum for datum in data if datum.Price >= 10 and datum.VolumeInUsd]” to “filtered = [datum for datum in data if datum.Price > 0 and datum.VolumeInUsd]”, the Coarse should return a full pair list (I tried Binance) which should have at least several hundred pairs, but the list returned by this demo code only has 29 pairs.
• If change this filtered code to “filtered = [datum for datum in data if datum.VolumeInUsd > 1000000]”, I got the following Error Message:

  Runtime Error: '>' not supported between instances of 'NoneType' and 'int' at filtered = [datum for datum at data if datum.VolumeInUsd > 1000000]

Tony    

Louis Szeto 

I am now receiving these coins as the top15 volume. This seems highly implausible

def UniverseSelectionFilter(self, data):

	# weekly universe selection
	if self.Time.weekday() != 6:
		return self.symbols
	filtered = [datum for datum in data if not datum.VolumeInUsd is None and str(datum.Symbol.Value).endswith("USDT")]
	filtered = [datum for datum in filtered if not datum.VolumeInUsd>=10000000]
	sorted_by_volume_in_usd = sorted(filtered, key=lambda datum: datum.VolumeInUsd, reverse=True)[0:15]
	self.symbols=[datum.Symbol for datum in sorted_by_volume_in_usd]
	self.Log(f"{self.Time} UNIVERSE={str([str(e.Value) for e in self.symbols])}")
	return self.symbols

Santa24 : I think that is to be expected because, on line #7, You're *filtering out* symbols that have VolumeInUsd over 10000000.

Removing that line will give you the usual high volume symbols (BTC, ETH, etc).

.ekz. oh yes you are right - sorry about that, my mistake, So for binance this ended up working for weekly universe selection top 10 by volume:

def UniverseSelectionFilter(self, data):
	# weekly universe selection
	if self.Time.weekday() != 6:
		return self.symbols
	filtered = [s for s in data if not s.VolumeInUsd is None and str(s.Symbol.Value).endswith("USDT")]
	filtered = [s for s in filtered if  str(s.Symbol.Value) != "USDCUSDT"]
	sorted_by_volume_in_usd = sorted(filtered, key=lambda s: s.VolumeInUsd, reverse=True)[0:10]
	self.symbols=[s.Symbol for s in sorted_by_volume_in_usd]
	return self.symbols

Hi Tony Gu

As we mentioned, although most situations will have this attribute available, some pairs in some unique days will not. It is recommended to check with:

[x for x in crypto_coarse if x.VolumeInUsd and x.VolumeInUsd > 100000]

Best
Louis

Hi Louis,

Thank you for following up on this question. It works.

Tony

This is great, thanks!

I am having trouble using a margin account with this framework. If I change the brokerage model AccountType from Cash to Margin, it still fails to execute my shorts with the message: 

• Backtest Handled Error: Unable to compute order quantity of LTCUSD E3. Reason: The cash model does not allow shorting. Returning null.

However, I find that if I comment out the below, short orders are accepted (but I don't want to do this, as it results in other problems with the script).

• # Warm up the security with the last known price to avoid conversion error        self.SetSecurityInitializer(lambda security: security.SetMarketPrice(self.GetLastKnownPrice(security)));

Any suggestions appreciated. Thanks! 

Hi Tom

We cannot reproduce the problem, do you mind attaching some code snippets or a backtest to let us understand?

Best
Louis

Hi Louis,

I've tried to use CryptoCoarseFundamentalUniverse in backtest. However, there is error message in CloudTerminal and the message is as followed.

Backtest Handled Error: We could not fetch the requested data. This may not be valid data, or a failed download of custom data. Skipping source (/Data/crypto/binance/fundamental/coarse/20220221.csv).

May I know if this is normal? Thanks in advance.

Hi Napoleon,

We are aware of the issue and have contacted the data vendor to patch the missing dates of Binance data.

Hi Louis.

I was hoping you could help me out. I cloned your sample code, and then tried to refactor the universe selection into its own class. However I am running into different errors depending on how I attempt to refactor the code.. I can't attach a backtest since they are failing to compile. So I am inserting the code here. 

Here is the main python class

from AlgorithmImports import *
from CryptoUniverseSelectionModel import CryptoUniverseSelectionModel

class CoinAPIDataAlgorithm(QCAlgorithm):

    def Initialize(self):
        self.SetStartDate(2020, 6, 1)
        self.SetEndDate(2020, 7, 1)
        self.SetCash(100000)
        self.SetCash("BTC", 1000)

        # Kraken accepts both Cash and Margin type account.
        self.SetBrokerageModel(BrokerageName.Kraken, AccountType.Margin)

        # Warm up the security with the last known price to avoid conversion error
        self.SetSecurityInitializer(lambda security: security.SetMarketPrice(self.GetLastKnownPrice(security)))
        
        self.UniverseSettings.Resolution = Resolution.Daily
        # Add universe selection of cryptos based on coarse fundamentals
        self.SetUniverseSelection(CryptoUniverseSelectionModel())
        
        self.AddAlpha(ConstantAlphaModel(InsightType.Price, InsightDirection.Up, timedelta(days=1), 0.025, None))
        self.SetPortfolioConstruction(EqualWeightingPortfolioConstructionModel())

    def OnSecuritiesChanged(self, changes):
        for security in changes.AddedSecurities:
            # Historical data
            history = self.History(security.Symbol, 30, Resolution.Daily)
            self.Debug(f"We got {len(history)} items from our history request")

Here is the CryptoUniverseSelectionModel  class

#region imports
from AlgorithmImports import *


# Your New Python File
class CryptoUniverseSelectionModel(CryptoCoarseFundamentalUniverse):

    def UniverseSelectionFilter(self, crypto_coarse):
        return [d.Symbol for d in sorted(crypto_coarse, key=lambda x: x.VolumeInUsd, reverse=True)[:5]]

The project does not compile since the method signature does not match and it throws the following error:

During the algorithm initialization, the following exception has occurred: Trying to dynamically access a method that does not exist throws a TypeError exception. To prevent the exception, ensure each parameter type matches those required by the () method. Please checkout the API documentation.
  at Initialize
    self.SetUniverseSelection(CryptoUniverseSelectionModel())
   at Python.Runtime.PythonException.ThrowLastAsClrException()
   at Python.Runtime.PyObject.Invoke(PyTuple args in main.py: line 19
 No method matches given arguments for .ctor: ()

If I try it by passing the parameters in so I can match the method signature 

#region imports
from AlgorithmImports import *


# Your New Python File
class CryptoUniverseSelectionModel(CryptoCoarseFundamentalUniverse(Market.Kraken, self.UniverseSettings, self.UniverseSelectionFilter)):

    def UniverseSelectionFilter(self, crypto_coarse):
        return [d.Symbol for d in sorted(crypto_coarse, key=lambda x: x.VolumeInUsd, reverse=True)[:5]]

 Then the project fails to compile with the following error

During the algorithm initialization, the following exception has occurred: Loader.TryCreatePythonAlgorithm(): Unable to import python module ./cache/algorithm/project/main.pyc. AlgorithmPythonWrapper(): name 'self' is not defined
  at 
    class CryptoUniverseSelectionModel(CryptoCoarseFundamentalUniverse(Market.Kraken in CryptoUniverseSelectionModel.py: line 6
  at 
    from CryptoUniverseSelectionModel import CryptoUniverseSelectionModel
   at Python.Runtime.PythonException.ThrowLastAsClrException()
   at Python.Runtime.NewReferenceExtensions.BorrowOrThrow(NewReference& reference)
   at Python.Runtime.PyModule.Import(String name)
   at Python.Runtime.Py.Import(String name)
   at QuantConnect.AlgorithmFactory.Python.Wrappers.AlgorithmPythonWrapper..ctor(String moduleName) at AlgorithmFactory/Python/Wrappers/AlgorithmPythonWrapper.cs:line 74 in main.py: line 1
 name 'self' is not defined

Would you by any chance be able to show me how to properly refactor the code so that the Universe selection lives in its own class? Thanks for your help.

Cheers,
Mitch