Everybody and their brother is holding themselves out as "the XBRL quality expert" for SEC XBRL financial filings these days. Well, here is some information you can use to get to the bottom of exactly how good someone's quality really is.
I have been analyzing SEC XBRL financial filings for years and years. Back in about November 2009, I found exactly one SEC XBRL financial filing which passed a battery of tests which I was using. By March 2010 the number that passed my battery of tests was 92. By about August of 2013 I had tuned my battery of tests and documented the things that I was looking at and the results I found in this Summary of Analysis of SEC XBRL Financial Filings document.
So, my tests were getting more tuned and harder. Tests are expanding toward accounting and financial reporting semantics or the meaning of the information. And at the same time the number of filings that passed the tests is growing. This is a very good sign.
These tests are not perfect by any means, nor are they comprehensive. But, if filers do not pass these tests, there is little hope that their SEC XBRL financial filing will be useful to anyone. However, all this said; there are thousands of additional tests which must be passed to prove that an XBRL-based financial filing is totally correct. My point is, I am not saying any SEC XBRL financial filing is totally correct.
All that said, quality is definitely improving. I have taken my tests, tuned them again, and ran them against the most current 10-K filings. My new set contains 5262 filings currently. The submission analyzed were 10-K SEC XBRL financial filing submissions between March 1, 2013 and December 31, 2013. This will be updated for 10-K filings submitted between January 1 and February 28, 2014.
This is my battery of tests: (All Stars pass all of these tests)
- XBRL technical syntax validation: ZERO XBRL technical syntax errors are allowed.
- Automated EFM validation: ZERO EDGAR Filer Manual (EFM) errors are allowed.
- US GAAP Taxonomy Architecture rules: ZERO US GAAP Taxonomy Architecture model structure errors are allowed. (For more information see this blog post.)
- Core financial statement roll ups: ZERO missing core financial statement roll ups (assets, liabilities and equity, net income, net cash flow) are allowed. (For more information, see this blog post.)
- Fundamental accounting concepts validation: ZERO failed fundamental accounting concept business rules. (For more information, see this blog post.) Note that this is pretty aggressive but this is necessary for unambiguous use of the financial information.
Below are the complete results thus far. I will be adding additional semantic tests relating to identification of the root reporting entity and correct reporting period information over the coming months. In the very near future I will also be adding additional accounting/financial reporting semantic rules. But as of right now, this is what the data says:
For the total set of 5,262 SEC XBRL financial filings, all of which are 10-Ks; I found 709 which passed all the tests in my battery of tests. That is an average of 13% of all SEC XBRL financial filings. In the list of generators, all generators who were average or better than average are highlighted in green.
I will update this information and make all my data available after the 10-K season which ends March 1, 2014. I will recompile the results, further tune my tests, and see what shows up.
If any software vendor, filing agent, SEC filer, or anyone else has any input as to what else might be included in my battery of tests please contact me.
Again, I am not saying that I am defining what quality is. You can decide for yourself what the definition of quality is. However, you should be able to justify your definition of quality. So should your software vendor or filing agent.