Testing Types Ontology v1.0.1907.1

Please find below a checklist I've created to describe different testing strategies.

1/ The hidden rationale behind the word "testing"

 • Testing vs. Checking
◇ [ ] "Testing" is explorative, probing and learning oriented.
◇ [ ] "Checking" is confirmative (verification and validation of what we already know). The outcome of a check is simply a pass or fail result; the outcome doesn't require human interpretation. Hence checking should be the first target for automation.

REMARK: We can see that speaking of “UnitTests” is actually not the best thing we came up with. We should rather be saying “UnitChecks”. But never mind.

2/ High level characteristics

• Interfaces
◇ [ ] GUI (Requires a dedicated Desktop Session)
◇ [ ] CLI-style (Tests can be run in parallel within a single User Session)
◇ [ ] or Hybrid CLI+GUI (Makes extensive use of CLI and access to the Desktop resource is leveraged through locking or distribution)

• Scopes
◇ [ ] Unit Test (Method testing, Mock testing)
◇ [ ] Component Tests (e.g. API Tests)
◇ [ ] System Tests (e.g. Integration Tests, Coded UI Tests)

• Areas of Concern
◇ [ ] Functional
◇ [ ] Acceptance
◇ [ ] Security
◇ [ ] Performance
▪ [ ] Load Testing: measure quantitative load.
▪ [ ] Stress Testing: check behavior under exceptional circumstances.

• Backends
◇ [ ] None (i.e. the tests generate their own data from scratch, dynamically),
◇ [ ] Snapshot-driven (w/ Upgrade of existing on-site data)
◇ [ ] Import-driven (e.g. using an import feature to perform testing on legacy or fresh data)
◇ [ ] or Hybrid

• Mechanics
◇ [ ] Data-driven (the same scenario is repeated over and over again, each time with different data)
◇ [ ] Scenario-based (each test requires a dedicated piece of software, typically a script)
◇ [ ] Exploratory (e.g. Bug Bounty, Hallway Testing, Usability Testing, Monkey Testing)

• Dependencies
◇ [ ] Application (depends on another process)
◇ [ ] Input Data (depends on flat files or streams of data)
◇ [ ] or Hybrid

• Privileges
◇ [ ] Local Administrator (eg. The tests requires write access to Program Files)
◇ [ ] Custom Local (eg. The tests access protected local resources)
◇ [ ] Any Local Standard User (Accesses a limited range of resources, attached drives, or only C:\Users\[UserName] directories.)

3/ Practical categories

• Operating
◇ [ ] Revert-based (each session starts from a well-known state, typically a virtual server snapshot)
◇ [ ] or Rolling (when nothing goes wrong, the testing server may have an ever increasing uptime)

 • Scheduling
◇ [ ] Triggered (e.g. Continuous Integration)
◇ [ ] Scheduled (e.g. Regular/Nightly)
◇ [ ] OnDemand (more suitable for very heavy, long and expensive tests – e.g. Load testing, Security testing, etc.)

• Reporting
◇ [ ] Fully Automated (PASS/FAIL)
◇ [ ] Computer Assisted (WARNINGS)
◇ [ ] Manual (Analytical)

4/ Volatile categories

The categories below are fairly volatile: the tests may move to a different category as the system-under-test grows older.

• Branches
◇ [ ] Internal/Staging
◇ [ ] Release

• Stages
◇ [ ] Development Tests (No backend/legacy data)
◇ [ ] Non-Regression Tests (Snapshot or Import-driven, see 'Backends' above.)

Bots and automated commits – Continuous Integration for Quality Assurance and Performance

The idea of having bots writing code is only the continuation of code analysis. In this mindset, 1/ one creates levels of abstractions and let automated systems write the actual code (eg. through snippets, custom DSLs, etc.) and 2/ one associates its code with tests that are run on automated systems and let the "bots" auto-merge features when tests pass.

For those who are interested, this information comes from the following article (a very long, but interesting thing to read btw): "Why Google Stores Billions of Lines of Code in a Single Repository | July 2016 | Communications of the ACM" - https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext

Code Analysis for C# – Comparing Visual Studio and Sonar Capabilities

I have made some attempts to run code analysis tools on C# code and I have a couple remarks and examples.

But first, let's clarify what is the technology that makes it so easy to inspect static C# code: the Roslyn compiler.

Roslyn doesn't have a preference for any particular set of rules. The "Rules" are just a plugin. Each of us could write our own set of rules if we wanted to (see "Tutorial: Write your first analyzer and code fix | Microsoft Docs" ).

Visual Studio provides its own implementation of a C# analyzer with its own set of rules. But as it turns out, based on my case study, those rules are not the same ones as the ones implemented and enforced in Sonar for instance. The default Visual Studio analyzer and the default Sonar analyzer are not chasing after the same things. In fact either of this tool might report as a "warning", things that the other does not report at all. I believe that the same thing applies to NDepend.

This means that tools like *Sonar* and *NDepend* are NOT just GUIs to a well established set of rules. Each tool comes with its own philosophy and system of beliefs about what is good or bad practice.

You might then say: whatever then, who's better than Microsoft for setting rules about the language they created, right? Maybe. And maybe not.

In practice, different analyzers are complementary. The analyzer provided by Visual Studio seems better at picking up edge cases related to the language, whereas Sonar seems to operate at a higher (logical) level.

---

EXAMPLE #1:

Example of a problem picked up by the the Visual Studio analyzer but not by by the Sonar analyzer :

the following code



Stream data = webclient.OpenRead(url);
StreamReader reader = new StreamReader(data);
xmlstring = reader.ReadToEnd();
data.Close();
reader.Close();



supposedly will cause Dispose() to be called twice on the 'data' object...

The following code removes the warnings:



using (StreamReader reader = new StreamReader(webclient.OpenRead(url)))
    xmlstring = reader.ReadToEnd();



Like I said, this was not picked up by Sonar.

---

EXAMPLE #2:

Example of a problem picked up by the the Sonar analyzer but not by by the Visual Studio analyzer :
"Comparing to itself always returns true."

FYI, I've turned on all info and warnings and made them all visible. Visual Studio just does not see the pb with `listPrevious.SequenceEqual(listPrevious)`  :stuck_out_tongue:  Which is a shame, because that is a real bug in production code (a typo, or a bad copy-paste), it's not something I just made up. And Sonar's set of rule was able to detect this.


So in conclusion, my feeling is that Visual Studio's set of rules is better at detecting the "edge cases" related to the actual syntax features of the language and the .NET APIs, (which is only natural since both are Microsoft's creations). Whereas Sonar, being 10 years old, and being born in another world (Java) and supporting many languages, is more capable when it comes to reasoning on the semantic and logical levels.

---

CONCLUSION:



If I had to rephrase, I think that Sonar just doesn't bother with the flavor of the language; it cares about the logic and the semantics. Whereas Visual Studio tries to make it hard to shoot yourself in the foot with this one particular language; but on the other hand, it will let you write silly code if you want to, as long as you respect the internal constructs and overall code design.

---

READ MORE:


To close on this matter, a final quantitative analysis gives us:

SONAR:

• Number of active C# rules in Sonar, by default = 216
• Maximum number of available C# rules in Sonar = 355

VISUAL:

• Number of active C# rules in Visual, by default = 62
• Maximum number of available C# rules in Visual = 455

RulesCount-inSonar:

RulesCount-inVisual:

RulesCount-inVisual+ALL:

REMARK 1: A shared server is the preferred way to use Sonar, because it is very easy to create projects (one – or more – per developer) and use individual tokens to upload results before checking code in (and browsing results anonymously, in read-only mode, within the company's LAN/VPN). [However, if really wanted, an individual, private localhost instance setup, is possible too, in about 20 minutes when you have a concise documentation.]

REMARK 2: There seems to be a Visual Studio plugin that allows to run Sonar's rule on the fly

SonarLint for Visual Studio 2017: https://marketplace.visualstudio.com/items?itemName=SonarSource.SonarLintforVisualStudio2017

SonarLint for Visual Studio 2019 : https://marketplace.visualstudio.com/items?itemName=SonarSource.SonarLintforVisualStudio2019

From the website:

SonarLint spots bugs and quality issues as fast as you code.

• 5 languages supported: C#, VB .Net, C, C++ and Javascript.
• Open source, Roslyn based code analyzers.
• Deep code analysis algorithms using pattern matching and dataflow analysis
• Hundreds of rules, and growing.
• Comes with explanations to resolve detected issues.

A word about false-positive and false-negative test-results, and why having a 'negative test-result' means 'passing the test'

Both “false-positive” and “false-negative” test-results exist.

1/ Positive and Negative tests-results

First, notice that the usual phrasing is “one is taking a test”, or “one is being tested for something”. One is “positive” for anomalies just like one is “positive” for alcohol, narcotics, drugs, disease, pregnancy, driving speed limit or amounts of goods at the customs: ie. typically some output values are above or below average. In everyday life, tests usually measure the concentration of a given chemical compound, or goods, or speeds; it’s the same for software: except that instead of chemical compounds or goods, we have file system objects, and we also have durations, speeds, loads, etc.

So, when software is taking a test, software is tested “for” regressions. Just like a chemical-test might reveal the presence of a molecule in your blood, a regression-test might reveal some incoherent or wrong behavior coming from the system-under-test. Therefore a “positive” test-result is a test-result that causes the test to appears, in first instance, and without further investigation, as “failed”.

By symmetry, a “negative” test-result is a result that did not detect any anomalies; the test is then usually considered as “passed”.

2/ False test-results

A “false-positive” test is a test that first appeared as “failed” but was later proven to be insignificant (eg. just like you would be first tested positive for something after saliva, or urine test, and later dismissed upon running more precise, blood tests).

Finally, “false-negative” tests are much more rare in practice: those are tests that did not catch an anomaly but should have. One can tell a test is a false-negative whenever one finds a problem that was not reported in the test covering the use-case.

In a nutshell:

Positive test-result = Anomaly detected = Failed test

Negative test-result = No anomaly detected = Passed test

False-positive test-result = Anomaly detected but no problem found after further investigation.

False-negative test-result = No anomaly detected and yet there was a bug.

Use another processor for heavy tasks... (Remote desktop connection)

Compiling can be a very annoying task: it takes ages and it uses all your CPU resources; it's especially true when your software project contains a few hundred thousands lines of code and that you have to perform a "Rebuild All" compilation.

Well, that was until today... because from now on, the "Rebuild All" doesn't run on my computer anymore... thanks to "Remote desktop connection"!

You can find this utility among the programs shipped with your Microsoft Windows XP PRO SP2 Operating System (and above):



Remote Desktop Connection allows you to connect to another computer using its address (IP or local network name):



If you want the other computer to work on your local data you may have to share your drive C:\

Warning: of course, you must trust your local network very very very much to do this! Or find another way...

Now, assuming that your computer's name is "\\MyComputer", you can make the second computer work on your data; just map the network drive using "Explorer > Tools > Map network drive":



And here you go... you can have an other processor doing the heavy tasks for you: