Digital Financial Reporting

The following is a summary of verification/validation results for a set of 7,199 SEC XBRL financial filings.  All filings are 10-Ks for the fiscal year 2012 filed with the SEC between March 1, 2012 and February 28, 2013.

Here is some information helpful in understanding these validation/verification results:

• XBRL technical syntax. This information comes from the XBRL Cloud EDGAR Dashboard. Basically 99.9% of all SEC XBRL financial filings in the set of filings I looked at were valid per the XBRL technical syntax specification.  What is a little odd is that there were 8 filings which were not valid.  What that means is that there are some XBRL interoperability issues.  The XBRL validator(s) used by the SEC for inbound validation did not detect the 8 filings as not being valid against the XBRL specification. XBRL Cloud believes that there are XBRL technical syntax errors in these 8 filings.  I validated each one against another XBRL processor and sure enough, I found errors in each of the 8 filings reported by that XBRL processor also.  As such, I would concur with XBRL Cloud's assessment. (I also posted a message to the XBRL Specification Working Group informing them of these errors.  Hopefully a conformance suite test will be added for each of the offending errors, which would improve consistency between XBRL processors.
• Edgar Filer Manual (EFM) automatible rules. Again, this information comes from XBRL Cloud. Why would XBRL Cloud's EFM validation differ from SEC EFM validation?  Mainly because of different interpretation of the rules by software vendors (i.e. the ones with the errors), the SEC, and XBRL Cloud. It would be very helpful if all software vendors and the SEC reported the same consistent results.  This is getting better, but still has a ways to go given that almost 20% of all filings have EFM rule violations.  And these are only the automatable rules.
• US GAAP Taxonomy Architecture Model Structure. These rules relate to the relationships between [Table]s, [Axis], [Member]s, [Line Items], [Abstract]s, and Concepts within an XBRL taxonomy.  They also relate to a degree to differences (i.e. ambiguity) between presentation, calculation, and definition relations. These relations are explained in the US GAAP Taxonomy Architecture (see section 4.5, Implementation of Tables).  Of all the 10-K filers, 98% follow these rules.
• US GAAP Domain Level Rules.  These are a small set of rules (i.e. there are many more) which every SEC XBRL financial filing must follow: assets must be reported, liabilities and equity must be reported, assets = liabilities and equity, equity must be reported, net income (loss) must be reported, and net cash flow must be reported. Why are these rules true?  Well, given that 96% of all filers follow these rules and that every filing which I looked at clearly did NOT follow the rule (but should have); it is pretty safe to say that all filers should pass these rules.  That is what "domain level" means, US GAAP specifies these sorts of things.  Ever hear of the accounting equation:  Assets = Liabilities + Equity?
• Balance sheet roll up rules missing. Balance sheets have "assets" and "liabilities and equity".  Balance sheets also balance.  "Assets" adds up, or "rolls up".  Same for "Liabilities and equity".  93% of all filers provide business rules in the form of XBRL calculations which are used to prove that the roll ups are working correctly.  7% did not.  If the rules are provided, then automated processes can be used to prove that the line items of the balance sheet roll up.  If they are not provided, you can only resort to manual error to make sure things add up. A clue as to which process might work best: automated processes save time and money.
• Income statement roll up rules missing. As with the balance sheet, the income statement rolls up.  It foots.  Part of the reason why there are so many missing XBRL calculation relations is that filers put numbers in their XBRL filings backwards because they try to get the polarity (positive or negative) to match the presentation.  I speculate that they give up in frustration or believe that they just cannot get them to add up correctly. A clue that you CAN make these add up is that about 80% of all filers seem to be able to provide these XBRL calculation rules.
• Cash flow statement roll up rules missing. Again, same deal for the cash flow statement; the line items add up and filers seem to be having problems with the polarity.

This is just a small set of the automatable rules which one can test for.  Why would you do that, seems like more work?  Well, it is actually less work.  Automating the testing of correct modeling, computations, and other rules and relations makes it easier to prove to yourself that you did a good job creating your SEC XBRL financial filing.

As the automated rules grows the quality of SEC XBRL financial filings will improve. Why?  Click here.

I believe that I am seeing my first definitive correlation between the quality of an SEC XBRL financial filing and the software used to create that filing.

What I did was look at all 7,199 SEC XBRL 10-K filings which I have been analyzing to try and find how many filers provide XBRL calculations for their balance sheets.  All SEC filers provide balance sheets. All balance sheets balance, that is "the accounting equation".  The XBRL Cloud Edgar Dashboard shows that this is true (see the column Domain Level Rules).

Well, balance sheets also foot.  Assets foot.  Liabilities and equity foot.  And so I ran a test against all 7,199 SEC XBRL financial filings, the 10-Ks in my set, and this is what I found:

• Of the 7,199; a total of 6,679 (92.8%) do provide XBRL calculations for the assets and liabilities & equity roll ups.
• Of the 7,199; a total of 520 (7.2%) do not.

But wait, this is the interesting part.  There seems to be a pattern in the software used to create SEC XBRL financial filings.  This is a breakdown of the same information by software used to create the SEC XBRL financial filing: (This Excel spreadsheet has the detailed results.)

There are two important things that I see in this data.  First, notice all the software vendors with zero missing XBRL calculations relations.  That means consistency.  Nothing falls through the cracks.

The second thing to notice is the high rates for a number of filers as compared to the rest of the pack.  Reach your own conclusions about what this means.  What it means to me is this: Why would there be any statistical correlation between whether an SEC filer needs business rules expressed and the software used to create the SEC XBRL financial filing?

This is, I believe, the first clear evidence that I have seen which shows a correlation between software and SEC XBRL financial filing quality.  I specuate that there will be more such correlations revealing themselves in the future as people dig into the filing information more.

What do you think?

What I think is that this is another clue that a financial report quality model will emerge.

I cherry picked from a set of 7,199 SEC XBRL financial filings, all of which are 10-Ks filed for fiscal years ended in 2012 (approximately).  You can get to this data set here in HTML, or you can download the data in Excel here.

My data set includes 2,213 filings, about 31% of the total.  The criteria which I set for picking the filings were as follows: (these criteria were arbitrarily set my me to improve the chance of all the numeric relations both working and PROVING to myself that they work using automated processes)

• The filer had to have core financial integrity correct meaning that I had to be able to find assets, liabilities and equity, equity, net income, and net cash flows in their filing.
• The filer is a commercial and industrial company, meaning that they report current assets and current liabilities.
• The filer reported total liabilities.
• The filer reported cash flows from operations, investing, and financing activities.
• The balance sheet balances.
• On the balance sheet, liabilities + commitments and contingencies + temporary equity + equity = assets.
• On the cash flow statement, operating cash flows + investing cash flows + financing cash flows + cash flows from discontinued operations + exchange gains/losses = net cash flow.
• On the income statement, income from continuing operations before taxes - income taxes + income/loss from discontinued operations + extraordinary items = net income.

The 2,213 filings that you see on the HTML page and in the Excel spreadsheet made that cut. There are links to both the SEC filing and to the XBRL Cloud Free Viewer so you can go look at the financial statements if you want.

What does this mean? Well, I am simply experimenting so I don't know that there are any definitive conclusions; but I think that the fact that I can pull all this information from those filings is a positive thing.

Granted, I cherry picked the SEC filers.  This is the "happy path" through the data.  I am avoiding filings which have errors or which have odd reporting situations.  I am sure I can get more filings which pass all the rules, for example if I look at filers who report but don't use a classified balance sheet, I speculate I would get a longer list. (Well, in fact, just to check I ran this query and I got 3,272 filers which do not have classified balance sheets but to pass all the other rules; that is 46% of all filers.)

One area where I am not able to effectively pull information from is the income statement above the line "income from continuing operations before taxes".  I will look into that next perhaps.  If I can do that I will be able to get high level financial information from the balance sheet, income statement, and cash flow statement. Again, I am not using an XBRL processor.  My tool is simply a Microsoft Access database.

Play with the data.  If you find anything interesting let me know.  If you have any ideas for experiments to run, let me know that also.