A Quantitative Trading Model with Stock Screener

Select the key elements to make decision

Here I choose 5 elements including roe, gross profit margin, liabilities, net profit cash flow, payout ratio to analyze stocks. The reason is not within the scope of this post. Finding out all of these data by hand will be a huge work and will be time-consuming. There exists some better ways to speed up this process. Of course, this is the purpose of my writing this post.
The stocks we will analyze mainly sold in many different stock markets, such as Shanghai Stock Exchange, Shenzhen Stock Exchange, Hong Kong Exchange, NASDAQ, New York Stock Exchange, American Stock Exchange, etc. We should filter the qualified stocks judged by the important elements we choose. There are a couple of methods to achieve that. This trading model doesn’t contain complicated calculation and most of the work can be done by stock screener sites and tools like excel.
There are tons of stock screeners on the Internet. The first thing is to select a useful one, it depends on yourself.

https://www.tradingview.com/screener/ https://finance.yahoo.com/screener/ https://www.investing.com/stock-screener/

You may need to grab Details of some information from many different sites. For instance, the dividend payout ratio can be found at https://finbox.com/, and you even need to pay for the data at other sites. For Apple:

https://finbox.com/AAPL/explorer/payout_ratio https://finance.yahoo.com/quote/AAPL/key-statistics/

Maybe some sites can be crawled by simple spiders. But most of the requests sent by the spiders will be banned by those sites. Of course, if we add some additional measures we can bypass those tricks. To deal with the interception of high-frequency requests, the easiest way is to set a time delay. Besides, you can even modify the header to pretend the real requests generated by the browsers. For JavaScript encrypted contents, you can read the code and find out your bypass ways. That will be a little bit troublesome. There are other tools like selenium, phantomjs, puppeteer to handle it. And Puppeteer will be a good choice in most situations, it also provides a python module named pyppeteer.

Be careful with pyppeteer

The current version of pyppeteer is not stable, it comes from the limitation of python, some asynchronous methods cannot be used in pyppeteer. And I spent considerable time on those confusing problems. For the exception thrown by pyppeteer:

pyppeteer.errors.NetworkError: Protocol Error (Runtime.callFunctionOn): Session closed. Most likely the page has been closed.

The proposal offered in https://github.com/miyakogi/pyppeteer/pull/160/files might be useful, but it doesn’t merge to the latest version, you need to modify the pyppeteer source code by yourself.
Sometimes the browser will crash because of small memory space, especially in a container like Docker or something like CI environment. What you want to do is to launch the browser with some arguments like this:

browser = await launch({'args': ['--disable-dev-shm-usage']})

The toughest problem I have encountered during the programming is the browser will always be stuck at page.goto step, finally the browser will be closed automatically. I guess the program might have triggered some scripts that will keep running and never come to an end to deal with the headless browsers. This is a big problem and I’m struggling on it from days to days. In the JavaScript version, puppeteer provides a couple of ways to stop the page loading and return all of the content. But I found a good way to implement the same function in the end, to use finally expression.

Take a look at the source code of page.goto, this method will raise an exception when it reaches the timeout limit.

def _createTimeoutPromise(self) -> Awaitable[None]:
    self._maximumTimer = self._loop.create_future()
    if self._timeout:
        errorMessage = f'Navigation Timeout Exceeded: {self._timeout} ms exceeded.'

        async def _timeout_func() -> None:
            await asyncio.sleep(self._timeout / 1000)
            self._maximumTimer.set_exception(TimeoutError(errorMessage))

        self._timeout_timer: Union[asyncio.Task, asyncio.Future] = self._loop.create_task(_timeout_func())

Our goal is to handle this exception and stop it from shutting down the program. It can be solved in such an easy way, so funny.

browser = await launch()
page = await browser.newPage()
try:
    await page.goto(url, timeout=10000)
except Exception as e:
    print(e)
finally:
    ...

The key to the problem is to stop the browser from closing. It may be not so convenient, but it’s robust enough to guarantee the necessary content can be completely fetched without shutting down.

Output

The source code can be found at https://github.com/recursively/quantitative_trading_pub. If you select the American stock market, the final output will be like this:

德州仪器 TXN
ROE: [57.73 35.39 35.21 29.37 26.62]
cashflow_profit：[1.44 1.47 1.28 1.46 1.29]
gross_profit：[65.11 64.26 61.63 58.15 56.93]
debt_ratio：[47.52 41.41 36.26 38.72 40.19]
bonus_ratio：[0.45860215 0.57088539 0.4581363  0.48392498 0.47075505]

迪士尼 DIS
ROE: [27.97 21.23 21.39 18.73 16.6 ]
cashflow_profit：[1.3  1.36 1.4  1.37 1.13]
gross_profit：[44.94 45.04 46.09 45.94 45.88]
debt_ratio：[45.28 51.82 48.58 44.82 42.74]
bonus_ratio：[0.20400064 0.28028953 0.26067511 0.26604629 0.25996534]

赛灵思 XLNX
ROE: [21.18 24.42 21.18 24.17 22.06]
cashflow_profit：[1.28 1.25 1.35 1.5  1.6 ]
gross_profit：[70.21 69.85 69.65 70.18 68.8 ]
debt_ratio：[53.36 47.07 45.99 46.19 44.66]
bonus_ratio：[0.70898438 0.55216693 0.59891107 0.48765432 0.47460317]

The program will calculate the appropriate price, the result below is from HKEX.

IGG 00799       Stock code: HK.00799 Last price:     5.75 gprice:     9.81

More to mention

Some implement in the source code needs to be modified to improve the performance. There are too many IO operations in the program, some functions can be replaced by the asynchronous method. Such as the extract_bonus function:

def extract_bonus(self, stock_code):
        res = requests.get(self.bonus_url.format(stock_code), headers=config.headers)
        root = etree.HTML(res.content)
        years = root.xpath('//*[@id="bonus_table"]/tbody/tr[*]/td[1]/text()')
        index = []
        recent_ratio = []
        for idx, val in enumerate(years):
            if "年报" in val:
                index.append(idx)

        ratio = root.xpath('//*[@id="bonus_table"]/tbody/tr[*]/td[9]/text()')
        for idx, val in enumerate(index):
            if idx < 5:
                recent_ratio.append(ratio[val])

Moreover, the browser just requests a single page for every launch. It wastes too much time during the whole procedure.

async def extract_debts(self, stock_code):
        browser = await launch({'headless': True, 'args': ['--disable-dev-shm-usage']})
        page = await browser.newPage()
        await page.goto(self.debt_url.format(stock_code), {'waitUntil': "networkidle2"}, timeout=60000)
        await page.waitForSelector('#cwzbTable')
        await page.click('#cwzbTable > div.scroll_container > ul > li:nth-child(2) > a')

        all_targets = await page.xpath('//*[@id="cwzbTable"]/div[1]/div[1]/div[4]/table[2]/tbody/tr[11]/td[position()<6]')
        for item in all_targets:
            self.debt_ratio.append(await (await item.getProperty('textContent')).jsonValue())

        await browser.close()

It can be replaced by requesting every page from a new tab instead of restarting the browser.