Testing Types Ontology v1.0.1907.1

Please find below a checklist I've created to describe different testing strategies.

1/ The hidden rationale behind the word "testing"

 • Testing vs. Checking
◇ [ ] "Testing" is explorative, probing and learning oriented.
◇ [ ] "Checking" is confirmative (verification and validation of what we already know). The outcome of a check is simply a pass or fail result; the outcome doesn't require human interpretation. Hence checking should be the first target for automation.

REMARK: We can see that speaking of “UnitTests” is actually not the best thing we came up with. We should rather be saying “UnitChecks”. But never mind.

2/ High level characteristics

• Interfaces
◇ [ ] GUI (Requires a dedicated Desktop Session)
◇ [ ] CLI-style (Tests can be run in parallel within a single User Session)
◇ [ ] or Hybrid CLI+GUI (Makes extensive use of CLI and access to the Desktop resource is leveraged through locking or distribution)

• Scopes
◇ [ ] Unit Test (Method testing, Mock testing)
◇ [ ] Component Tests (e.g. API Tests)
◇ [ ] System Tests (e.g. Integration Tests, Coded UI Tests)

• Areas of Concern
◇ [ ] Functional
◇ [ ] Acceptance
◇ [ ] Security
◇ [ ] Performance
▪ [ ] Load Testing: measure quantitative load.
▪ [ ] Stress Testing: check behavior under exceptional circumstances.

• Backends
◇ [ ] None (i.e. the tests generate their own data from scratch, dynamically),
◇ [ ] Snapshot-driven (w/ Upgrade of existing on-site data)
◇ [ ] Import-driven (e.g. using an import feature to perform testing on legacy or fresh data)
◇ [ ] or Hybrid

• Mechanics
◇ [ ] Data-driven (the same scenario is repeated over and over again, each time with different data)
◇ [ ] Scenario-based (each test requires a dedicated piece of software, typically a script)
◇ [ ] Exploratory (e.g. Bug Bounty, Hallway Testing, Usability Testing, Monkey Testing)

• Dependencies
◇ [ ] Application (depends on another process)
◇ [ ] Input Data (depends on flat files or streams of data)
◇ [ ] or Hybrid

• Privileges
◇ [ ] Local Administrator (eg. The tests requires write access to Program Files)
◇ [ ] Custom Local (eg. The tests access protected local resources)
◇ [ ] Any Local Standard User (Accesses a limited range of resources, attached drives, or only C:\Users\[UserName] directories.)

3/ Practical categories

• Operating
◇ [ ] Revert-based (each session starts from a well-known state, typically a virtual server snapshot)
◇ [ ] or Rolling (when nothing goes wrong, the testing server may have an ever increasing uptime)

 • Scheduling
◇ [ ] Triggered (e.g. Continuous Integration)
◇ [ ] Scheduled (e.g. Regular/Nightly)
◇ [ ] OnDemand (more suitable for very heavy, long and expensive tests – e.g. Load testing, Security testing, etc.)

• Reporting
◇ [ ] Fully Automated (PASS/FAIL)
◇ [ ] Computer Assisted (WARNINGS)
◇ [ ] Manual (Analytical)

4/ Volatile categories

The categories below are fairly volatile: the tests may move to a different category as the system-under-test grows older.

• Branches
◇ [ ] Internal/Staging
◇ [ ] Release

• Stages
◇ [ ] Development Tests (No backend/legacy data)
◇ [ ] Non-Regression Tests (Snapshot or Import-driven, see 'Backends' above.)

Code Analysis for C# – Comparing Visual Studio and Sonar Capabilities

I have made some attempts to run code analysis tools on C# code and I have a couple remarks and examples.

But first, let's clarify what is the technology that makes it so easy to inspect static C# code: the Roslyn compiler.

Roslyn doesn't have a preference for any particular set of rules. The "Rules" are just a plugin. Each of us could write our own set of rules if we wanted to (see "Tutorial: Write your first analyzer and code fix | Microsoft Docs" ).

Visual Studio provides its own implementation of a C# analyzer with its own set of rules. But as it turns out, based on my case study, those rules are not the same ones as the ones implemented and enforced in Sonar for instance. The default Visual Studio analyzer and the default Sonar analyzer are not chasing after the same things. In fact either of this tool might report as a "warning", things that the other does not report at all. I believe that the same thing applies to NDepend.

This means that tools like *Sonar* and *NDepend* are NOT just GUIs to a well established set of rules. Each tool comes with its own philosophy and system of beliefs about what is good or bad practice.

You might then say: whatever then, who's better than Microsoft for setting rules about the language they created, right? Maybe. And maybe not.

In practice, different analyzers are complementary. The analyzer provided by Visual Studio seems better at picking up edge cases related to the language, whereas Sonar seems to operate at a higher (logical) level.

---

EXAMPLE #1:

Example of a problem picked up by the the Visual Studio analyzer but not by by the Sonar analyzer :

the following code



Stream data = webclient.OpenRead(url);
StreamReader reader = new StreamReader(data);
xmlstring = reader.ReadToEnd();
data.Close();
reader.Close();



supposedly will cause Dispose() to be called twice on the 'data' object...

The following code removes the warnings:



using (StreamReader reader = new StreamReader(webclient.OpenRead(url)))
    xmlstring = reader.ReadToEnd();



Like I said, this was not picked up by Sonar.

---

EXAMPLE #2:

Example of a problem picked up by the the Sonar analyzer but not by by the Visual Studio analyzer :
"Comparing to itself always returns true."

FYI, I've turned on all info and warnings and made them all visible. Visual Studio just does not see the pb with `listPrevious.SequenceEqual(listPrevious)`  :stuck_out_tongue:  Which is a shame, because that is a real bug in production code (a typo, or a bad copy-paste), it's not something I just made up. And Sonar's set of rule was able to detect this.


So in conclusion, my feeling is that Visual Studio's set of rules is better at detecting the "edge cases" related to the actual syntax features of the language and the .NET APIs, (which is only natural since both are Microsoft's creations). Whereas Sonar, being 10 years old, and being born in another world (Java) and supporting many languages, is more capable when it comes to reasoning on the semantic and logical levels.

---

CONCLUSION:



If I had to rephrase, I think that Sonar just doesn't bother with the flavor of the language; it cares about the logic and the semantics. Whereas Visual Studio tries to make it hard to shoot yourself in the foot with this one particular language; but on the other hand, it will let you write silly code if you want to, as long as you respect the internal constructs and overall code design.

---

READ MORE:


To close on this matter, a final quantitative analysis gives us:

SONAR:

• Number of active C# rules in Sonar, by default = 216
• Maximum number of available C# rules in Sonar = 355

VISUAL:

• Number of active C# rules in Visual, by default = 62
• Maximum number of available C# rules in Visual = 455

RulesCount-inSonar:

RulesCount-inVisual:

RulesCount-inVisual+ALL:

REMARK 1: A shared server is the preferred way to use Sonar, because it is very easy to create projects (one – or more – per developer) and use individual tokens to upload results before checking code in (and browsing results anonymously, in read-only mode, within the company's LAN/VPN). [However, if really wanted, an individual, private localhost instance setup, is possible too, in about 20 minutes when you have a concise documentation.]

REMARK 2: There seems to be a Visual Studio plugin that allows to run Sonar's rule on the fly

SonarLint for Visual Studio 2017: https://marketplace.visualstudio.com/items?itemName=SonarSource.SonarLintforVisualStudio2017

SonarLint for Visual Studio 2019 : https://marketplace.visualstudio.com/items?itemName=SonarSource.SonarLintforVisualStudio2019

From the website:

SonarLint spots bugs and quality issues as fast as you code.

• 5 languages supported: C#, VB .Net, C, C++ and Javascript.
• Open source, Roslyn based code analyzers.
• Deep code analysis algorithms using pattern matching and dataflow analysis
• Hundreds of rules, and growing.
• Comes with explanations to resolve detected issues.