之前在回顾一个几个月前参加的某个Kaggle比赛时，翻到了一个非常有趣的异常点检测notebook。链接在这里https://www.kaggle.com/snippsy/feature-importance-over-time-for-outlier-detection

这个方法适用于时间序列数据。

我当时稍微想了想，发现这个方法特么可以推广到财务数据上来啊。然后懒癌的关系又拖了一个多月，今天花了点时间完工了，效果感觉还可以。现在写下来介绍一下。

涉及模型：ExtraTreesRegressor、IsolationForest

涉及平台：米筐量化平台、同花顺问财

算法假设：整一套算法是基于公司每个年份对标签预测的特征重要性是无太大波动的这一假设。

流程简要：在对财务数据预处理后，构建标签，然后按年份将数据扔到ExtraTreesRegressor里面去并输出特征重要性。将所有年份的特征重要性集合起来做成新数据再扔到IsolationForest里面去，最后模型输出哪些年份属于异常年份。

步骤：

以下部分代码属于米筐量化平台才可运行。

---

一、导包

import pandas as pd
import numpy as np
from sklearn.ensemble import IsolationForest,ExtraTreesRegressor
from tqdm import tqdm

二、导入行业分类数据

这里我选择的是同花顺问财的行业分类数据，感觉比米筐的好些。数据可以从同花顺问财下载。

ind = pd.read_csv('ind.csv',encoding='gbk')[['股票代码','所属同花顺行业']]

三、米筐财务数据字段

Profit = ['revenue','operating_revenue','net_interest_income','net_commission_income','commission_income','commission_expense','net_proxy_security_income','sub_issue_security_income','net_trust_income','earned_premiums','premiums_income','reinsurance_income','reinsurance','unearned_premium_reserve','total_expense','operating_expense','refunded_premiums','compensation_expense','amortization_expense','premium_reserve','amortization_premium_reserve','policy_dividend_payout','reinsurance_cost','other_operating_revenue','other_operating_cost','r_n_d','other_net_income','net_open_hedge_income','other_revenue','gredit_asset_impairment','o_n_a_expense','amortization_reinsurance_cost','insurance_commission_expense','disposal_income_on_asset','cost_of_goods_sold','sales_tax','gross_profit','selling_expense','ga_expense','financing_expense','financing_interest_income','financing_interest_expense','exchange_gains_or_losses','profit_from_operation','invest_income_associates','fair_value_change_income','investment_income','asset_impairment','interest_income','interest_expense','non_operating_revenue','non_operating_expense','disposal_loss_on_asset','other_effecting_total_profits_items','profit_before_tax','income_tax','unrealised_investment_loss','other_effecting_net_profits_items','net_profit','non_recurring_pnl','net_profit_deduct_non_recurring_pnl','classified_by_continuity_operation','continuous_operation_net_profit','discontinued_operation_net_profit','classified_by_ownership','net_profit_parent_company','minority_profit','other_income','other_income_unclassified_income_statement','remearsured_other_income','other_income_equity_unclassified_income_statement','other_equity_instruments_change',
         'corporate_credit_risk_change','other_income_classified_income_statement','other_income_equity_classified_income_statement','financial_asset_available_for_sale_change','financial_asset_hold_to_maturity_change','cash_flow_hedging_effective_portion','foreign_currency_statement_converted_difference','others','other_debt_investment_change','assets_reclassified_other_income','other_debt_investment_reserve','other_income_minority','total_income','total_income_parent_company','total_income_minority','basic_earnings_per_share','fully_diluted_earnings_per_share']
Balnce = ['financial_asset_held_for_trading','cash_equivalent','client_deposits','bill_receivable','dividend_receivable','bill_accts_receivable','interest_receivable','bad_debt_reserve','net_accts_receivable','contract_assets','prepayment','financial_receivable','financial_lease_receivable','prepaid_tax','other_equity_investment','other_illiquidy_financial_assets','non_current_asset_due_one_year','other_receivables_interest_dividend','inventory','consumable_biological_assets','deferred_expense','assets_hold_for_sale','contract_work','other_current_assets','current_assets','financial_asset_available_for_sale','non_current_liability_due_one_year','debt_investment','other_debt_investment','financial_asset_hold_to_maturity','real_estate_investment','long_term_equity_investment','long_term_receivables','net_long_term_equity_investment','accumulated_depreciation','depreciation_reserve','net_fixed_assets','total_fixed_assets','engineer_material','construction_in_progress','total_construction_in_progress','fixed_asset_to_be_disposed','capitalized_biological_assets','oil_and_gas_assets','intangible_assets','seat_costs','impairment_intangible_assets','use_right_assets','goodwill','long_term_deferred_expenses','deferred_income_tax_assets','other_non_current_assets','non_current_assets','loan_account_receivables','fund_providing','reinsurance_reserve_receivable','settlement_provision','client_provision','interbank_deposits','precious_metals','lend_capital','derivative_financial_assets','resale_financial_assets','loans_advances_to_customers','insurance_receivable','subrogation_fee_receivable','reinsurance_receivable','unearned_reserve_receivable','unclaimed_reserve_receivable','life_reserve_receivable','health_reserve_receivable','insurer_mortgage_loan','fixed_deposits','refundable_deposits','refundable_capital_deposits','independent_account_assets','other_assets','other_accts_receivable','total_assets','mortgaged_loan','short_term_loans','financial_liabilities','notes_payable','accts_payable','bill_accts_payable','contract_liabilities','advance_from_customers','payroll_payable','dividend_payable','tax_payable','interest_payable','other_fees_payable','other_payable','other_payable_interest_dividend','short_term_debt','accrued_expense','liabilities_hold_for_sale','estimated_liabilities','deferred_income','long_term_liabilities_due_one_year','other_current_liabilities','current_liabilities','long_term_loans','bond_payable','preference_shares','perpetual_bond','long_term_payable','accrued_staff_costs','grants_received','housing_revolving_funds',
         'deferred_income_tax_liabilities','lease_liabilities','financial_lease_payable','other_non_current_liabilities','non_current_liabilities','borrowings_from_central_banks','deposits_of_interbank','borrowings_capital','derivative_financial_liabilities','buy_back_security_proceeds','deposits','proxy_security_proceeds','sub_issue_security_proceeds','security_deposits_received','advance_insurance','comission_payable','reinsurance_payable','compensation_payable','policy_dividend_payable','deposits_from_interbank','insurance_contract_reserve','insurer_deposit_investment','uncertained_premium_reserve','unclaimed_indemnity_reserve','life_insurance_reserve','health_insurance_reserve','independent_account_liabilities','other_liabilities','provision','deferred_revenue','total_liabilities','paid_in_capital','other_equity_instruments','equity_preferred_stock','perpetual_equity_debt','capital_reserve','surplus_reserve','undistributed_profit','treasury_stock','equity_parent_company','total_equity','general_reserve','trade_risk_allowances','foreign_currency_converted_difference','uncertained_impairment_losses','other_reserves','specific_reserve','minority_interest','total_equity_and_liabilities']
Cash = ['cash_received_from_sales_of_goods','refunds_of_taxes','net_deposit_increase','net_increase_from_central_bank','net_increase_from_other_financial_institutions','draw_back_canceled_loans','cash_received_from_interests_and_commissions','net_increase_from_disposing_financial_assets','net_increase_from_repurchasing_business','cash_received_from_original_insurance','cash_received_from_reinsurance','net_increase_from_insurer_deposit_investment','net_increase_from_financial_institutions','cash_received_from_proxy_security','cash_received_from_sub_issue_security','cash_from_other_operating_activities','cash_from_operating_activities','cash_paid_for_goods_and_services','assets_depreciation_reserves','exchange_rate_change_effect','other_effecting_cash_equivalent_items','cash_equivalent_increase','begin_period_cash_equivalent','end_period_cash_equivalent','cash_paid_for_employee','cash_paid_for_taxes','net_increase_from_loans_and_advances','net_increase_from_central_bank_and_banks','net_increase_from_lending_capital','cash_paid_for_comissions','cash_paid_for_orignal_insurance','cash_paid_for_reinsurance','cash_paid_for_policy_dividends','net_increase_from_trading_financial_assets','net_increase_from_operating_buy_back',
       'cash_paid_for_other_operation_activities','cash_paid_for_operation_activities','cash_flow_from_operating_activities','cash_received_from_disposal_of_investment','cash_received_from_investment','cash_received_from_disposal_of_asset','cash_received_from_other_investment_activities','cash_received_from_investment_activities','cash_paid_for_asset','cash_paid_to_acquire_investment','cash_paid_for_other_investment_activities','cash_paid_for_investment_activities','cash_flow_from_investing_activities','cash_received_from_investors','cash_received_from_minority_invest_subsidiaries','cash_received_from_issuing_security','cash_received_from_financial_institution_borrows','cash_received_from_issuing_equity_instruments','net_increase_from__financing_buy_back','cash_received_from_other_financing_activities','cash_received_from_financing_activities','cash_paid_for_debt','cash_paid_for_dividend_and_interest','dividends_paid_to_minority_by_subsidiaries','cash_paid_for_other_financing_activities','cash_paid_to_financing_activities','cash_flow_from_financing_activities','net_cash_deal_from_sub','net_cash_payment_from_sub','net_increase_in_pledge_loans','net_increase_from_investing_buy_back','net_inc_cash_and_equivalents','fixed_asset_depreciation','deferred_expense_amortization','intangible_asset_amortization']

四、按财报时间读取数据、预处理、构建标签

1、读取数据

年报公布时间最晚是在每年的4月30日，因此我们可以将当年5月1日到次年4月30日作为一个年份。

2、数据预处理

由于财报数据对模型来说太少了，所以要想办法扩容。最好的办法就是仿照市盈率公式，用公司总市值/各个财报科目。对于NaN和inf值，均用0填充。

（米筐对每个财报因子都是输出当季度的值。比如一个财报年，米筐会输出四个值。）

3.构建标签

这是一个可以优化的地方，如果想到有更好的标签可以有更好的模型效果。我目前用的是【每日目标股票的涨跌幅-每日目标股票所属行业的所有股票的涨跌幅均值】。以此来衡量当日目标股票偏离其所属行业多少。ExtraTreesRegressor模型任务是预测次日偏离值。

code = '600077.XSHG' #这里填股票代码
year = range(2010,2021,1)
st = '0501'
ed = '0430'
industry = ind.loc[ind['股票代码']==Sstr(code),'所属同花顺行业'].values[0]
ind_stock = ind.loc[ind['所属同花顺行业']==industry,'股票代码'].apply(Bstr)
new_data = pd.DataFrame()
for y in year:
    start = str(y)+st
    end = str(y+1)+ed
    label = get_price_change_rate(code,start,end)-get_price_change_rate(ind_stock,start,end).mean(axis=1)
    pro = get_factor(code,Profit,start,end).fillna(0)
    bal = get_factor(code,Balnce,start,end).fillna(0)
    cas = get_factor(code,Cash,start,end).fillna(0)
    temp_data = pd.concat([pro,bal,cas],axis=1).reset_index()
    temp_data = temp_data.drop(['order_book_id','date'],axis=1)
    cap = get_factor(code,'market_cap_3',start,end).fillna(0).to_numpy()
    temp_data = pd.DataFrame(np.nan_to_num(temp_data.apply(lambda x: cap/x,axis=0),posinf=0,neginf=0),columns=temp_data.columns)
    temp_data['date'] = y
    temp_data['label'] = label.values
    temp_data['label'] =temp_data['label'].shift(-1)
    temp_data = temp_data.dropna(axis=0,how='any')
    new_data = new_data.append(temp_data)

五、模型训练

这一步主要是要按不同年份进行训练（每个年份都会训练一个模型），同时将整个算法运行十次，最后将异常年份进行出现次数的统计。

time_list = new_data['date'].drop_duplicates().to_list()
fea_imp = []
res = pd.Series()
tt = 0
while tt<10:
    for t in tqdm(time_list):
        new = new_data.loc[new_data['date']==t,:].copy()
        new = new.drop('date',axis=1)
        lab = new.pop('label')
        etc = ExtraTreesRegressor()
        etc.fit(new,lab)
        fea_imp.append(etc.feature_importances_)
    feature_importance = np.asarray(fea_imp)
    pred = IsolationForest().fit_predict(feature_importance)
    outlier_days = [day for day,pred in zip(time_list,pred) if pred < 0]
    res = res.append(pd.Series(outlier_days))
    fea_imp = []
    tt+=1
print(res.value_counts())

六、实践

我们用这篇文章里的部分公司进行试验：https://www.wogoo.com/essay/#/forwarding?articleId=e08334e42a1a40f79b408c9a1fd8d93b&articleType=0

时间从2010年-2020年。

$*ST金正(002470.SZ)$ 

2020    10
2019     6
2017     3
2010     3
dtype: int64

$宋都股份(600077.SH)$ 

2011    10
2018     7
2020     6
2019     3
2010     3
dtype: int64

$康普顿(603798.SH)$ （康普顿16年上市，所以开始日期为20160501）

2020    2
2019    2
dtype: int64

$万润股份(002643.SZ)$ （万润股份11年底上市，所以开始日期为20120501）

2020    10
2014     6
2015     4
2016     1
dtype: int64

七、算法缺陷和能改进的地方

1.这个算法不能辨别某年份的财报异常是属于积极的还是消极的。

2.标签的构建可能存在更好的选择。

3.米筐不能提供一些更详细的财务数据，理论上说，更好的财务特征能够带来更好的提升。